Sitelet https://github.com/ethereum/go-ethereum/pull/17914
Skip to content

core/vm, eth: add standard json tracing into filesystem dumps - #17914

Merged
karalabe merged 2 commits into
ethereum:masterfrom
holiman:block_analysis
Dec 10, 2018
Merged

core/vm, eth: add standard json tracing into filesystem dumps#17914
karalabe merged 2 commits into
ethereum:masterfrom
holiman:block_analysis

Conversation

@holiman

@holiman holiman commented Oct 15, 2018 •
edited
Loading

Copy link
Copy Markdown
Contributor

This PR adds debug methods to dump out traces from blocks and/or individual transactions.

When @karalabe implemented tracing, the intended usecase was to enable long-running tracers that could stream results back via a subscription channel. However, when investigating e..g consensus errors and/or bad blocks, that format is not very good.

  • The existing generic tracer framework collects all result data (per block) into a big blob in memory, before streaming. For a large trace, such as the 205K steps in the recent Ropsten-fork, the full output from the tracer was 2.4GB, which can easily crash many nodes.

  • The existing callTracer does not confirm to std-json, which is

    • jsonl not json, to allow for streaming output
    • Uses op for string-representation of opcode, instead of op/opName for numeric/string, and other simlar small differences.

This PR enables dumping a tx from a given block:

> debug.standardTraceBlockToFile("0x0bbe9f1484668a2bf159c63f0cf556ed8c8282f99e3ffdb03ad2175a863bca63", {txHash:"0x4049f61ffbb0747bb88dc1c85dd6686ebf225a3c10c282c45a8e0c644739f7e9", disableMemory:true})
["/tmp/block_0x0bbe9f14-14-0x4049f61f-099048234"]

Or all txs from a block:

> debug.standardTraceBlockToFile("0x0bbe9f1484668a2bf159c63f0cf556ed8c8282f99e3ffdb03ad2175a863bca63", {disableMemory:true})
["/tmp/block_0x0bbe9f14-0-0xb4502ea7-409046657", "/tmp/block_0x0bbe9f14-1-0xe839be8f-954614764", "/tmp/block_0x0bbe9f14-2-0xc6e2052f-542255195", "/tmp/block_0x0bbe9f14-3-0x01b7f3fe-209673214", "/tmp/block_0x0bbe9f14-4-0x0f290422-320999749", "/tmp/block_0x0bbe9f14-5-0x2dc0fb80-844117472", "/tmp/block_0x0bbe9f14-6-0x35542da1-256306111", "/tmp/block_0x0bbe9f14-7-0x3e199a08-086370834", "/tmp/block_0x0bbe9f14-8-0x87778b88-194603593", "/tmp/block_0x0bbe9f14-9-0xbcb081ba-629580052", "/tmp/block_0x0bbe9f14-10-0xc254381a-578605923", "/tmp/block_0x0bbe9f14-11-0xcc434d58-405931366", "/tmp/block_0x0bbe9f14-12-0xce61967d-874423181", "/tmp/block_0x0bbe9f14-13-0x05a20b35-267153288", "/tmp/block_0x0bbe9f14-14-0x4049f61f-606653767", "/tmp/block_0x0bbe9f14-15-0x46d473d2-614457338", "/tmp/block_0x0bbe9f14-16-0x35cf5500-411906321", "/tmp/block_0x0bbe9f14-17-0x79222961-278569788", "/tmp/block_0x0bbe9f14-18-0xad84e7b1-095032683", "/tmp/block_0x0bbe9f14-19-0x4bd48260-019097038", "/tmp/block_0x0bbe9f14-20-0x1517411d-292624085", "/tmp/block_0x0bbe9f14-21-0x6857e350-971385904", "/tmp/block_0x0bbe9f14-22-0xbe3ae2ca-236639695"]

Files are created in a temp-location, with the naming standard block_<blockhash:4>-<txindex>-<txhash:4>-<random suffix>. Each opcode immediately streams to file, with no in-geth buffering aside from whatever buffering the os normally does.

It also adds some more info when regenerating historical state, namely, the reexec-number if required historical state is not avaiable is encountered, so a user can experiment with increasing that setting. It also prints out the remaining block until it reaches target:

INFO [10-15|13:48:25.263] Regenerating historical state            block=2385959 target=2386012 remaining=53   elapsed=3m30.990537767s
INFO [10-15|13:48:33.342] Regenerating historical state            block=2386012 target=2386012 remaining=0    elapsed=3m39.070073163s
INFO [10-15|13:48:33.343] Historical state regenerated             block=2386012 elapsed=3m39.070454362s nodes=10.03mB preimages=652.08kB
INFO [10-15|13:48:33.352] Wrote trace                              file=/tmp/block_0x14490c57-0-0xfbbd6d91-715824834
INFO [10-15|13:48:33.352] Wrote trace                              file=/tmp/block_0x14490c57-1-0x71076194-187462969
INFO [10-15|13:48:34.421] Wrote trace file=/tmp/block_0x14490c57-2-0x3f4263fe-056924484

There are two new methods added:

debug.standardTraceBlockToFile(blockhash, options)
debug.standardTraceBadBlockToFile(blockhash, options)

Where options is

type StdTraceConfig struct {
	*vm.LogConfig
	Reexec *uint64
	TxHash *common.Hash
}

One more thing: it also changes the existing debug_traceBadBlock to take a block hash instead of index-of-badblocks. A hash is less ambiguous and more natural choice.

@holiman
holiman requested a review from karalabe as a code owner October 15, 2018 11:59
@holiman

holiman commented Oct 15, 2018

Copy link
Copy Markdown
Contributor Author

Travis only has a spurious swarm failure

@holiman holiman mentioned this pull request Oct 25, 2018
Closed
@karalabe karalabe self-assigned this Nov 7, 2018
@holiman

holiman commented Nov 28, 2018

Copy link
Copy Markdown
Contributor Author

Rebased to fix conflicts. PTAL

@karalabe karalabe added this to the 1.8.20 milestone Dec 10, 2018
@karalabe karalabe changed the title Better block/tx analysis core/vm, eth: add standard json tracing into filesystem dumps Dec 10, 2018
holiman
holiman commented Dec 10, 2018
View reviewed changes

@holiman holiman left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

karalabe
karalabe approved these changes Dec 10, 2018
View reviewed changes

@karalabe karalabe left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM too

@karalabe
karalabe merged commit fd66af5 into ethereum:master Dec 10, 2018
@holiman holiman mentioned this pull request Jan 11, 2019
Closed
@holiman holiman mentioned this pull request Jan 24, 2019
Merged
@holiman holiman mentioned this pull request Feb 8, 2019
Closed
@holiman holiman mentioned this pull request Feb 9, 2019
Closed
This was referenced Sep 12, 2019
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@karalabe karalabe karalabe approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@karalabe karalabe

Labels

None yet

Projects

None yet

Milestone

1.8.20

Development

Successfully merging this pull request may close these issues.

3 participants

@holiman @karalabe @adamschmideg