Panic on eth_call when copying the state of a pending block

[vitalii427]

System information

Geth version: 1.14.5-stable
CL client & version: Lighthouse v5.1.3-3058b96
OS & Version: Amazon Linux (4.14.314-237.533.amzn2.x86_64)
Commit hash: own tiny custom patch

Expected behaviour

We have a few nodes running as public RPCs on Mainnet. Our RPC nodes have a tiny custom patch that has not changed for the past few months. We serve quite a large number of requests.

Actual behaviour

Some nodes started to crash serving eth_call on June 7, crashing about once every day or two. The crash occurs when copying the state of a pending block. All requests that crash the node are targeted at the latest block and seem nothing special otherwise. Backtrace always looks the same.

Steps to reproduce the behaviour

I cannot effectively reproduce it with my own eth_call requests.

Backtrace

unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x429631]

goroutine 79441054 [running]:
runtime.throw({0x1a6a6d0?, 0x429194?})
	runtime/panic.go:1077 +0x5c fp=0xc04249e8c8 sp=0xc04249e898 pc=0x4514fc
runtime.sigpanic()
	runtime/signal_unix.go:875 +0x285 fp=0xc04249e928 sp=0xc04249e8c8 pc=0x468625
runtime.(*bmap).overflow(...)
	runtime/map.go:210
runtime.mapclone2(0x17d1fc0, 0xc0339008a0)
	runtime/map.go:1557 +0x3d1 fp=0xc04249e9c8 sp=0xc04249e928 pc=0x429631
maps.clone({0x17d1fc0, 0xc0339008a0})
	runtime/map.go:1454 +0x25 fp=0xc04249e9e8 sp=0xc04249e9c8 pc=0x481265
maps.Clone[...](...)
	maps/maps.go:46
github.com/ethereum/go-ethereum/trie.(*tracer).copy(0xc016b1bcb0)
	github.com/ethereum/go-ethereum/trie/tracer.go:102 +0x185 fp=0xc04249eac8 sp=0xc04249e9e8 pc=0xcfb165
github.com/ethereum/go-ethereum/trie.(*Trie).Copy(...)
	github.com/ethereum/go-ethereum/trie/trie.go:73
github.com/ethereum/go-ethereum/trie.(*StateTrie).Copy(0xc0214c17a0)
	github.com/ethereum/go-ethereum/trie/secure_trie.go:247 +0x26 fp=0xc04249eb48 sp=0xc04249eac8 pc=0xcf3ce6
github.com/ethereum/go-ethereum/core/state.(*cachingDB).CopyTrie(0xc0094d89d0?, {0x1ffb820?, 0xc0214c17a0?})
	github.com/ethereum/go-ethereum/core/state/database.go:216 +0x3d fp=0xc04249eba8 sp=0xc04249eb48 pc=0xd4c01d
github.com/ethereum/go-ethereum/core/state.(*stateObject).deepCopy(0xc01d68b260, 0xc02b7b04e0)
	github.com/ethereum/go-ethereum/core/state/state_object.go:529 +0x267 fp=0xc04249ec00 sp=0xc04249eba8 pc=0xd533a7
github.com/ethereum/go-ethereum/core/state.(*StateDB).Copy(0xc02e6b16c0)
	github.com/ethereum/go-ethereum/core/state/statedb.go:700 +0x616 fp=0xc04249eef8 sp=0xc04249ec00 pc=0xd57916
github.com/ethereum/go-ethereum/miner.(*Miner).Pending(0x0?)
	github.com/ethereum/go-ethereum/miner/miner.go:98 +0x29 fp=0xc04249ef18 sp=0xc04249eef8 pc=0xf49d09
github.com/ethereum/go-ethereum/eth.(*EthAPIBackend).StateAndHeaderByNumber(0xc192c6a5c93a2c9e?, {0x1fee9f8?, 0xc00bc73040?}, 0x1a856cb?)
	github.com/ethereum/go-ethereum/eth/api_backend.go:194 +0x38 fp=0xc04249ef90 sp=0xc04249ef18 pc=0x10631d8
github.com/ethereum/go-ethereum/eth.(*EthAPIBackend).StateAndHeaderByNumberOrHash(0x0?, {0x1fee9f8?, 0xc00bc73040?}, {0xc03e64eb38, 0x0, 0x0})
	github.com/ethereum/go-ethereum/eth/api_backend.go:217 +0x2ce fp=0xc04249f090 sp=0xc04249ef90 pc=0x106360e
github.com/ethereum/go-ethereum/internal/ethapi.DoCall({0x1fee9f8, 0xc00bc73040}, {_, _}, {0xc037a97fe0, 0xc02aef2018, 0x0, 0x0, 0x0, 0x0, ...}, ...)
	github.com/ethereum/go-ethereum/internal/ethapi/api.go:1157 +0x163 fp=0xc04249f240 sp=0xc04249f090 pc=0x102f3c3
github.com/ethereum/go-ethereum/internal/ethapi.(*BlockChainAPI).Call(0xc0bcecb870, {0x1fee9f8, 0xc00bc73040}, {0xc037a97fe0, 0xc02aef2018, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...)
	github.com/ethereum/go-ethereum/internal/ethapi/api.go:1177 +0x339 fp=0xc04249f3d8 sp=0xc04249f240 pc=0x102f959
runtime.call256(0xc0c39f1620, 0xc0c31d9888, 0xc025ecd1e0, 0xd0, 0xd0, 0x100, 0xc04249fa18)
	runtime/asm_amd64.s:751 +0x5e fp=0xc04249f4e8 sp=0xc04249f3d8 pc=0x4862fe
runtime.reflectcall(0x17bcde0?, 0x0?, 0x8?, 0x1a7d3aa?, 0x0?, 0x12?, 0x17bcde0?)
	<autogenerated>:1 +0x36 fp=0xc04249f528 sp=0xc04249f4e8 pc=0x48a576
reflect.Value.call({0xc006b1b700?, 0xc0c31d9888?, 0x7f9fc879a1f8?}, {0x1a69b6c, 0x4}, {0xc01804dcb0, 0x6, 0x42cc52?})
	reflect/value.go:596 +0xce7 fp=0xc04249fb38 sp=0xc04249f528 pc=0x4bd907
reflect.Value.Call({0xc006b1b700?, 0xc0c31d9888?, 0x0?}, {0xc01804dcb0?, 0x0?, 0x0?})
	reflect/value.go:380 +0xb9 fp=0xc04249fbb0 sp=0xc04249fb38 pc=0x4bc9d9
github.com/ethereum/go-ethereum/rpc.(*callback).call(0xc0c3256120, {0x1fee9f8?, 0xc00bc73040}, {0xc03e64eae8, 0x8}, {0xc030444540, 0x4, 0xc0065bdd20?})
	github.com/ethereum/go-ethereum/rpc/service.go:205 +0x37c fp=0xc04249fcd8 sp=0xc04249fbb0 pc=0xd88f5c
github.com/ethereum/go-ethereum/rpc.(*handler).runMethod(0xc00e3dcf70?, {0x1fee9f8, 0xc00bc73040}, 0xc02d190b60, 0x4?, {0xc030444540, 0x4, 0x4})
	github.com/ethereum/go-ethereum/rpc/handler.go:566 +0xe5 fp=0xc04249fd30 sp=0xc04249fcd8 pc=0xd82365
github.com/ethereum/go-ethereum/rpc.(*handler).handleCall(0xc0189e2460, 0xc020e4a450, 0xc02d190b60)
	github.com/ethereum/go-ethereum/rpc/handler.go:512 +0x22f fp=0xc04249fdd0 sp=0xc04249fd30 pc=0xd81a8f
github.com/ethereum/go-ethereum/rpc.(*handler).handleCallMsg(0xc0189e2460, 0xc020e4a540?, 0xc02d190b60)
	github.com/ethereum/go-ethereum/rpc/handler.go:470 +0x22d fp=0xc04249feb0 sp=0xc04249fdd0 pc=0xd813cd
github.com/ethereum/go-ethereum/rpc.(*handler).handleNonBatchCall(0xc0189e2460, 0xc020e4a450, 0xc02d190b60)
	github.com/ethereum/go-ethereum/rpc/handler.go:296 +0x187 fp=0xc04249ff50 sp=0xc04249feb0 pc=0xd7fa87
github.com/ethereum/go-ethereum/rpc.(*handler).handleMsg.func1.1(0x1fee9f8?)
	github.com/ethereum/go-ethereum/rpc/handler.go:269 +0x25 fp=0xc04249ff78 sp=0xc04249ff50 pc=0xd7f8e5
github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc.func1()
	github.com/ethereum/go-ethereum/rpc/handler.go:387 +0xbe fp=0xc04249ffe0 sp=0xc04249ff78 pc=0xd806be
runtime.goexit()
	runtime/asm_amd64.s:1650 +0x1 fp=0xc04249ffe8 sp=0xc04249ffe0 pc=0x487ae1
created by github.com/ethereum/go-ethereum/rpc.(*handler).startCallProc in goroutine 79441052
	github.com/ethereum/go-ethereum/rpc/handler.go:383 +0x79

[vitalii427]

"help wanted"

[karalabe]

You seem to be using an old version of Go. This bug golang/go#62203 seems to have been fixed in both 1.22.0 and 1.21.1

[karalabe]

Could you check if you're on Go 1.21.0 by any chance (AFAIK that was the only version with this bug)

[karalabe]

As for why this might have triggered now, we've received a number of PRs that kept replacing literal map copyings with these maps.Clone calls, so that might be the reason it started happening just now for you.

[vitalii427]

@karalabe Yes, I've used Go 1.21.0 and that seemed to be the issue