Debug_traceBlockByHash returning "insufficient funds" error on block 19888327

[Enigmatic331]

System information

Geth version: geth.exe version 1.14.3-stable-ab48ba42
CL client & version: beacon-chain version Prysm/v5.0.3/38f208d70dc95b12c08403f5c72009aaa10dfe2f. Built at: 2024-04-04 18:48:31+00:00
OS & Version: Windows

Expected behaviour

Debug_traceBlockByHash should return the trace dump for block 19888327

Actual behaviour

A tracing failed message was returned instead

C:\Users\Administrator>curl http://localhost:8545 -m 300 -X POST --header "Content-type: application/json" --data "{\"jsonrpc\":\"2.0\", \"method\":\"debug_traceBlockByHash\", \"params\":[\"0xcc1d35533ea26392bfa91944b3f6d1f446581b05f5f4b35dea93d558b08d353c\", {\"tracer\":\"callTracer\"}], \"id\":1}"
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"tracing failed: insufficient funds for gas * price + value: address 0xc79d40C897FbD912a7cD29E318a0D41F579793f6 have 2469004274997060 want 3963014826093176"}}

If I independently trace the two "out" transactions on this block for the address 0xc79d40C897FbD912a7cD29E318a0D41F579793f6 , it seems to return results correctly.

C:\Users\Administrator>curl http://localhost:8545 -m 300 -X POST --header "Content-type: application/json" --data "{\"jsonrpc\":\"2.0\", \"method\":\"debug_traceTransaction\", \"params\":[\"0xefa4345e670ef386c0c1e076625c1a7627df192adf5ed8b7f5a8ec27c14ccdfd\", {\"tracer\":\"callTracer\"}], \"id\":1}"
{"jsonrpc":"2.0","id":1,"result":{"from":"0xc79d40c897fbd912a7cd29e318a0d41f579793f6","gas":"0x927c0","gasUsed":"0x60126","to":"0x5f6ae08b8aeb7078cf2f96afb089d7c9f51da47d","input":"0xba1f77e80000000000000000000000000000000000000000000000000000000000000000","calls":[{... etc

C:\Users\Administrator>curl http://localhost:8545 -m 300 -X POST --header "Content-type: application/json" --data "{\"jsonrpc\":\"2.0\", \"method\":\"debug_traceTransaction\", \"params\":[\"0xb3552b1f6e4c97349ca45ae34bb831c04c571fd4fa2110481d4aae851e8a0331\", {\"tracer\":\"callTracer\"}], \"id\":1}"
{"jsonrpc":"2.0","id":1,"result":{"from":"0xc79d40c897fbd912a7cd29e318a0d41f579793f6","gas":"0xa410","gasUsed":"0x5208","to":"0x742c5ce550ec2e1a78b338904978ec9004eb065f","input":"0x","value":"0xc8ed706a30328","type":"CALL"}}

Seems to be affecting more than 1 of our nodes, potentially other blocks as well as we received some reports about indexed traces showing a transaction failing, however when the tx is traced individually with debug_traceTransaction, no execution reverted seemed to be found.

[holiman]

That block contains three transactions concerning that account

• OUT 0xb3552b1f6e4c97349ca45ae34bb831c04c571fd4fa2110481d4aae851e8a0331
• OUT 0xefa4345e670ef386c0c1e076625c1a7627df192adf5ed8b7f5a8ec27c14ccdfd
• IN 0xd5fd51583eab8fdbb885b017683ee9e5d6b2154103261c49a996cfc542b3dc23

[Enigmatic331]

The last tx could also run through debug_traceTransaction, though I didn't raise it as it didn't seem to concern 0xc79 as a sender 🙈 - Assuming that the insufficient funds for gas * price + value: address 0xc79d40C897FbD912a7cD29E318a0D41F579793f6 have 2469004274997060 want 3963014826093176 error targets address 0xc79 sending a transaction to a contract/EOA.

C:\Users\Administrator>curl http://localhost:8545 -m 300 -X POST --header "Content-type: application/json" --data "{\"jsonrpc\":\"2.0\", \"method\":\"debug_traceTransaction\", \"params\":[\"0xd5fd51583eab8fdbb885b017683ee9e5d6b2154103261c49a996cfc542b3dc23\", {\"tracer\":\"callTracer\"}], \"id\":1}"
{"jsonrpc":"2.0","id":1,"result":{"from":"0x742c5ce550ec2e1a78b338904978ec9004eb065f","gas":"0xa410","gasUsed":"0x5208","to":"0xc79d40c897fbd912a7cd29e318a0d41f579793f6","input":"0x","value":"0x15bc4cf4757ac0","type":"CALL"}}

[holiman]

So, looking at the IN one state changes:

Before that, the account has nothing. So it's not just a case of "the block-tracing doesn't commit the transaction state changes and executes every trace on the block prestate", because then the error would say have 0 want...

[Enigmatic331]

Ahhh you're right... Now where did the number 2469004274997060 came from...

[mtbitcoin]

Also block# 19888387 , #19888680 with similar issues if you need extra samples

[s1na]

Are you using the hash scheme or path scheme? If you first synced your node before v1.14.0 and didn't explicitly specify --state.scheme=path then it is hash scheme. v1.14.0 flips the default to path but only if you do a fresh sync.

[Enigmatic331]

We're using hash for our nodes (especially for this one where we are running archive nodes) 🙏

[quickchase]

Yeah, archive nodes have to be hash, path isn't supported, but this happens on either, I believe

[holiman]

I've been investigating this a bit, haven't found any smoking gun. The callTracer is a 'native' tracer, so it doesn't go into the more complicated parallel prestate-execution path, but only the simpler step by step tracing of transactions.

I have found one thing that looks quite odd, https://github.com/ethereum/go-ethereum/blob/master/eth/tracers/api.go#L985 , where we use a vmconfig with no basefee checking vm.Config{Tracer: tracer.Hooks, NoBaseFee: true}, and a mostly empty txcontext with e.g. zero gasprice vm.TxContext{GasPrice: big.NewInt(0)}. I don't see how that could be correct.

[holiman]

It seems, that earlier, it generated the txContext from the msg, but it was changed to the 'empty' form here:

064f37d#diff-bd288cd51c1874c7e65ce0d35047c0a98366479c8924606c9efc577f6a98f75cL947

@s1na any thoughts why?

EDIT: Nevermind, it seems that it's now set within ApplyTransactionWithEVM

	// Create a new context to be used in the EVM environment.
	txContext := NewEVMTxContext(msg)
	evm.Reset(txContext, statedb)

[celochoi]

I checked there was no problem with 1.13.15.
Does this issue only occur in 1.14.x?

[holiman]

@s1na the setting of NoBaseFee: true in combination with vm.TxContext{GasPrice: big.NewInt(0)}, will have consequences. Later on, during vm.NewEVM:

	if config.NoBaseFee {
		if txCtx.GasPrice.BitLen() == 0 {
			blockCtx.BaseFee = new(big.Int)
		}
		if txCtx.BlobFeeCap != nil && txCtx.BlobFeeCap.BitLen() == 0 {
			blockCtx.BlobBaseFee = new(big.Int)
		}
	}

That is: we will set the block ctx fields to zero. Thus overwriting the fields that we constructed earlier, in eth/tracers/api.go:

		blockCtx  = core.NewEVMBlockContext(block.Header(), api.chainContext(ctx), nil)

This will affect the opcodes BASEFEE and BLOBBASEFEE.
If anyone can provide me with the full opcode by opcode output of https://etherscan.io/vmtrace?txhash=0xefa4345e670ef386c0c1e076625c1a7627df192adf5ed8b7f5a8ec27c14ccdfd, that'd be greatly appreciated. I'd like to see what opcodes are executed.

[pengin7384]

@holiman
Did you mean debug_traceTransaction response?

Request

{
    "method": "debug_traceTransaction",
    "params": [
       "0xefa4345e670ef386c0c1e076625c1a7627df192adf5ed8b7f5a8ec27c14ccdfd"
    ],
    "id": 1,
    "jsonrpc": "2.0"
}

Response (w/ geth 1.13.15)
response.json