[Upgrade] Go-Ethereum release v1.9.21 #1211

quorumbot · 2021-06-09T13:24:24Z

TODO

Plan & Analyse

Review the Release Notes
Review PRs in the section below

As you review, list extra changes and/or tests to be implemented to ensure compatibility with GoQuorum specific features.

Build & Test

Pull and checkout PR branch locally, then merge GoQuorum master into this branch
Resolve conflicts, taking into account the prior analysis
Implement required changes until lint passes
Implement required changes until all unit tests pass
Implement required changes until acceptance tests pass
Implement extra changes and/or tests
Verify any left TODOs in the code

Extra Changes & Tests

debug_traceCall : it's a new JSON-RPC API method that allows to call a contract with different arguments than debug_traceTransaction. This has been adapted so that it is compatible with private contract. Link to the source code
did not upgraded leveldb and protobuf because of our version of etcd. These upgrades cause some acceptance tests failures. Link to the commit

Left TODOs

We should upgrade the etcd we use v3.3.13-quorum197+incompatible so that it is compatible with protobuf 1.4.2. It will unblock the usage of a newer leveldb version. We could align the versions with original go-ethereum
- Newer version of leveldb will fix a problem with released snapshots, have better performance, and a new configuration set. See PR

Go-Ethereum Release: Tavum (v1.9.21)

Version: v1.9.21
Published: 2020-09-09T08:25:11Z

Release notes

Geth v1.9.21 is a regular maintenance release, the highlights being the removal of whisper, better call tracing and multiple memory stability fixes during fast sync to both stabilize usage as well as to fix a memory leak that lead to crashers before.

Remove Whisper as promised a couple months ago (#21487, #21526, #21527)!
Minor user experience polishes around legacy Ledger derivation paths (#21517).
Implement arbitrary call tracing via debug_traceCall on top of arbitrary blocks (#21338).
Expose internal transaction revertals and revert reason in the call_tracer tracer (#21387).
Cap the number of in-memory trie nodes during fast sync, fix crasher memory leak (#21491).
Prepare the trie syncer for path-based operation to support the upcoming snap sync (#21504).
Limit the cached data in the downloader more aggressively to avoid memory fluctuations (#21366).
Fix the simulated chain to not allow changing block timestamps if transactions were included (#21334).
Fix an ABI parser issue around tuples (#21501).

For a full rundown of the changes please consult the Geth 1.9.21 release milestone

As with all our previous releases, you can find the:

Pre-built binaries for all platforms on our downloads page.
Docker images published under ethereum/client-go.
Ubuntu packages in our Launchpad PPA repository.
OSX packages in our Homebrew Tap repository.

Codebase changes assessment

Legend

File Stats: (A) Added, (M) Modified and (R) Removed

Line Stats: (A) Added and (R) Removed

Assessment:

✅ No conflict expected
⚠ Review required to assess changes
‼️ Conflicts expected and review required

19 Pull Requests

🔍	Link	Title	File Stats M/A/R	Packages changed (files changed)	Line Stats A/R	Top 5 Changed Files (lines changed)
‼️	#21334	`accounts/abi/bind/backends: Disallow AdjustTime for non-empty blocks`	2/0/0	`accounts/abi/bind/backends` (2)	44/5	`accounts/abi/bind/backends/simulated_test.go` (82) `accounts/abi/bind/backends/simulated.go` (16)
⚠️	#21490	`eth: updated comments`	3/0/0	`eth` (3)	3/3	`eth/api.go` (4) `eth/api_tracer.go` (4) `eth/bloombits.go` (4)
‼️	#21448	``go.mod	goleveldb latest update``	2/0/0	`go.mod` (1) `go.sum` (1)	54/17
✅	#21491	`core/state, eth, trie: stabilize memory use, fix memory leak`	7/0/0	`trie` (4) `core/state` (2) `eth/downloader` (1)	58/29	`trie/sync.go` (120) `eth/downloader/downloader.go` (22) `core/state/sync.go` (12) `trie/committer.go` (8) `core/state/statedb.go` (4)
⚠️	#21387	`eth/tracers: revert reason in call_tracer + error for failed internal calls`	3/2/0	`eth/tracers/internal/tracers` (2) `eth/tracers/testdata` (2) `eth/tracers` (1)	172/15	`eth/tracers/testdata/call_tracer_inner_instafail.json` (144) `eth/tracers/testdata/call_tracer_revert_reason.json` (128) `eth/tracers/tracers_test.go` (48) `eth/tracers/internal/tracers/call_tracer.js` (30) `eth/tracers/internal/tracers/assets.go` (24)
✅	#21497	`rpc: fix issue with null JSON-RPC messages`	3/0/0	`rpc/testdata` (2) `rpc` (1)	16/3	`rpc/json.go` (26) `rpc/testdata/invalid-batch.js` (6) `rpc/testdata/invalid-nonobj.js` (6)
✅	#21501	`accounts/abi: fix a bug in getTypeSize method`	2/0/0	`accounts/abi` (2)	37/1	`accounts/abi/type_test.go` (72) `accounts/abi/type.go` (4)
✅	#21503	`internal: fix personal.sign()`	1/0/0	`console` (1)	2/2	`console/bridge.go` (8)
✅	#21455	`"Downloader queue stats" is now provided once per minute`	1/0/0	`eth/downloader` (1)	1/1	`eth/downloader/queue.go` (4)
‼️	#21432	`go.mod: goja update`	2/0/0	`go.mod` (1) `go.sum` (1)	3/3	`go.sum` (8) `go.mod` (4)
✅	#21366	`eth/downloader: change intial download size`	5/0/0	`eth/downloader` (5)	26/24	`eth/downloader/downloader_test.go` (48) `eth/downloader/queue.go` (28) `eth/downloader/queue_test.go` (12) `eth/downloader/downloader.go` (8) `eth/downloader/testchain_test.go` (4)
✅	#21504	`core, eth, trie: prepare trie sync for path based operation`	6/0/0	`trie` (4) `core/state` (1) `eth/downloader` (1)	480/105	`eth/downloader/statesync.go` (306) `trie/sync_test.go` (298) `core/state/sync_test.go` (254) `trie/trie.go` (168) `trie/sync.go` (132)
⚠️	#21338	`eth: added trace_call to trace on top of arbitrary blocks`	3/0/0	`eth` (2) `internal/web3ext` (1)	54/8	`eth/api_tracer.go` (98) `eth/api.go` (14) `internal/web3ext/web3ext.go` (12)
‼️	#21514	`accounts/abi/bid/backends:`	2/0/0	`accounts/abi/bind/backends` (2)	76/76	`accounts/abi/bind/backends/simulated.go` (264) `accounts/abi/bind/backends/simulated_test.go` (40)
✅	#21487	`whisper: remove whisper`	6/0/24	`whisper/whisperv6` (17) `mobile` (3) `cmd/geth` (3) `whisper/mailserver` (2) `cmd/wnode` (1) `tests/fuzzers/whisperv6` (1) `whisper/shhclient` (1) `cmd/utils` (1) `tests/fuzzers/whisperv6/corpus` (1)	17/5826	`whisper/whisperv6/filter_test.go` (1672) `cmd/wnode/main.go` (1546) `whisper/whisperv6/api.go` (1186) `whisper/whisperv6/message_test.go` (942) `whisper/whisperv6/message.go` (710)
⚠️	#21517	`accounts/usbwallet, signer/core: show accounts from ledger legacy derivation paths`	2/0/0	`accounts/usbwallet` (1) `signer/core` (1)	30/17	`signer/core/api.go` (66) `accounts/usbwallet/wallet.go` (28)
⚠️	#21526	`build: remove wnode from the list of packages binaries`	1/0/0	`build` (1)	0/5	`build/ci.go` (10)
✅	#21527	`.github: remove whisper from CODEOWNERS`	1/0/0	`.github` (1)	0/1	`.github/CODEOWNERS` (2)
⚠️	#21534	`params: update CHTs for v1.9.21 release`	1/0/0	`params` (1)	16/16	`params/config.go` (64)

72 Changed files

🔍	File	Lines Changed	Linked PR
✅	`whisper/whisperv6/whisper.go`	2280
✅	`whisper/whisperv6/whisper_test.go`	1856
✅	`whisper/whisperv6/filter_test.go`	1672	#21487
✅	`cmd/wnode/main.go`	1546	#21487
✅	`whisper/whisperv6/api.go`	1186	#21487
✅	`whisper/whisperv6/message_test.go`	942	#21487
✅	`whisper/whisperv6/message.go`	710	#21487
✅	`whisper/whisperv6/envelope.go`	560	#21487
✅	`whisper/whisperv6/peer.go`	536	#21487
✅	`whisper/whisperv6/filter.go`	524	#21487
✅	`whisper/mailserver/server_test.go`	470	#21487
✅	`whisper/mailserver/mailserver.go`	418	#21487
✅	`whisper/whisperv6/benchmarks_test.go`	416	#21487
✅	`mobile/shhclient.go`	390	#21487
✅	`whisper/shhclient/client.go`	386	#21487
✅	`eth/downloader/statesync.go`	306	#21504
✅	`trie/sync_test.go`	298	#21504
‼️	`accounts/abi/bind/backends/simulated.go`	280	#21514 #21334
✅	`whisper/whisperv6/topic_test.go`	268	#21487
✅	`core/state/sync_test.go`	254	#21504
✅	`trie/sync.go`	240	#21491 #21504
✅	`mobile/types.go`	186	#21487
✅	`whisper/whisperv6/doc.go`	184	#21487
✅	`whisper/whisperv6/envelope_test.go`	182	#21487
✅	`tests/fuzzers/whisperv6/whisper-fuzzer.go`	180	#21487
✅	`whisper/whisperv6/gen_newmessage_json.go`	180	#21487
✅	`trie/trie.go`	172	#21491 #21504
✅	`whisper/whisperv6/gen_message_json.go`	168	#21487
✅	`eth/tracers/testdata/call_tracer_inner_instafail.json`	144	#21387
✅	`whisper/whisperv6/gen_criteria_json.go`	132	#21487
✅	`whisper/whisperv6/api_test.go`	128	#21487
‼️	`go.sum`	128	#21432 #21448
✅	`eth/tracers/testdata/call_tracer_revert_reason.json`	128	#21387
‼️	`accounts/abi/bind/backends/simulated_test.go`	122	#21514 #21334
✅	`whisper/whisperv6/topic.go`	112	#21487
‼️	`eth/api_tracer.go`	102	#21490 #21338
✅	`accounts/abi/type_test.go`	72	#21501
⚠️	`signer/core/api.go`	66	#21517
⚠️	`params/config.go`	64	#21534
✅	`whisper/whisperv6/config.go`	62	#21487
‼️	`cmd/utils/flags.go`	54	#21487
‼️	`cmd/geth/config.go`	54	#21487
⚠️	`eth/downloader/downloader_test.go`	48	#21366
⚠️	`eth/tracers/tracers_test.go`	48	#21387
✅	`eth/downloader/queue.go`	32	#21455 #21366
⚠️	`eth/downloader/downloader.go`	30	#21491 #21366
✅	`eth/tracers/internal/tracers/call_tracer.js`	30	#21387
⚠️	`accounts/usbwallet/wallet.go`	28	#21517
‼️	`go.mod`	26	#21432 #21448
⚠️	`rpc/json.go`	26	#21497
✅	`eth/tracers/internal/tracers/assets.go`	24	#21387
✅	`mobile/geth.go`	20	#21487
‼️	`eth/api.go`	18	#21490 #21338
‼️	`cmd/geth/consolecmd_test.go`	16	#21487
✅	`core/state/sync.go`	12	#21491
✅	`eth/downloader/queue_test.go`	12	#21366
✅	`trie/secure_trie.go`	12	#21504
⚠️	`internal/web3ext/web3ext.go`	12	#21338
⚠️	`build/ci.go`	10	#21526
✅	`trie/committer.go`	8	#21491
✅	`console/bridge.go`	8	#21503
✅	`rpc/testdata/invalid-nonobj.js`	6	#21497
✅	`rpc/testdata/invalid-batch.js`	6	#21497
⚠️	`eth/bloombits.go`	4	#21490
✅	`trie/trie_test.go`	4	#21491
⚠️	`cmd/geth/usage.go`	4	#21487
⚠️	`core/state/statedb.go`	4	#21491
✅	`accounts/abi/type.go`	4	#21501
✅	`eth/downloader/testchain_test.go`	4	#21366
⚠️	`params/version.go`	4
✅	`.github/CODEOWNERS`	2	#21527
✅	`tests/fuzzers/whisperv6/corpus/009c5adfa4fd685caef58e1ce932fa7fb209730a`	0	#21487

…(#21334) * accounts/abi/bind/backends: Disallow timeshift for non-empty blocks * accounts/abi/bind/backends: added tests for adjust time * accounts/abi/bind/simulated: added comments, fixed test for AdjustTime * accounts/abi/bind/backends: updated comment

* go.mod | goleveldb latest update * go.mod update * leveldb options * go.mod: double check Co-authored-by: Péter Szilágyi <[email protected]>

core/state, eth, trie: stabilize memory use, fix memory leak

… calls (#21387) * tests: add testdata of call tracer * eth/tracers: return revert reason in call_tracer * eth/tracers: regenerate assets * eth/tracers: add error message even if no exec occurrs, fixes #21438 Co-authored-by: Martin Holst Swende <[email protected]>

* accounts/abi: fix a bug in getTypeSize method e.g. for "Tuple[2]" type, the element of the array is a tuple type and the size of the tuple may not be 32. * accounts/abi: add unit test of getTypeSize method

* "Downloader queue stats" is now a DEBUG information I think this info is more a DEBUG related information then an INFO. If it must remains an INFO, maybe it can be slow down to one time every 5 minutes or so. * Update queue.go "Downloader queue stats" information is now provided once every minute instead of once every 10 seconds.

This changes how the downloader works, a little bit. Previously, when block sync started, we immediately started filling up to 8192 blocks. Usually this is fine, blocks are small in the early numbers. The threshold then is lowered as we measure the size of the blocks that are filled. However, if the node is shut down and restarts syncing while we're in a heavy segment, that might be bad. This PR introduces a more conservative initial threshold of 2K blocks instead.

core, eth, trie: prepare trie sync for path based operation

* eth: Added TraceTransactionPending * eth: Implement Trace_Call, remove traceTxPending * eth: debug_call -> debug_traceCall, recompute tx environment if pruned * eth: fix nil panic * eth: improve block retrieving logic in tracers * internal/web3ext: add debug_traceCall to console

…nd anonymous variable assignments (#21514) Co-authored-by: Osoro Bironga <[email protected]>

* whisper: remove whisper * Update cmd/geth/config.go Co-authored-by: Marius van der Wijden <[email protected]> * cmd/geth: warn on enabling whisper + remove more whisper deps * mobile: remove all whisper references Co-authored-by: Marius van der Wijden <[email protected]> Co-authored-by: Martin Holst Swende <[email protected]>

…ivation paths (#21517) * accounts/usbwallet, signer/core: un-hide accounts from ledger legacy derivation paths * Update accounts/usbwallet/wallet.go * Update signer/core/api.go * Update signer/core/api.go

params: update CHTs for v1.9.21 release

…125909

…cts without any signature change

…/go-ethereum/v1.9.21-2021609142421

nmvalera

LGTM

ricardolyn · 2021-06-15T10:55:55Z

go.mod

 	github.com/go-sourcemap/sourcemap v2.1.2+incompatible // indirect
 	github.com/go-stack/stack v1.8.0
 	github.com/golang/mock v1.4.3
 	github.com/golang/protobuf v1.3.4


should we add a comment here to maintain the old version?

do you mean add comment in order to keep the geth's version?
I prefer to prepare another PR to solve the remaining TODOs, which is a revert of the last commit.

@baptiste-b-pegasys can you create an issue to track this, so we are aware we need to put back the original version from geth?

in the meantime, I can merge this.

fjl and others added 25 commits August 25, 2020 16:21

params: begin v1.9.21 release cycle

fc20680

core/state, eth, trie: stabilize memory use, fix memory leak

d8da0b3

eth: updated comments (#21490)

16d7eae

go.mod | goleveldb latest update (#21448)

856307d

* go.mod | goleveldb latest update * go.mod update * leveldb options * go.mod: double check Co-authored-by: Péter Szilágyi <[email protected]>

Merge pull request #21491 from karalabe/state-sync-leak-fix

d97e006

core/state, eth, trie: stabilize memory use, fix memory leak

rpc: fix issue with null JSON-RPC messages (#21497)

5883afb

accounts/abi: fix a bug in getTypeSize method (#21501)

12d8570

* accounts/abi: fix a bug in getTypeSize method e.g. for "Tuple[2]" type, the element of the array is a tuple type and the size of the tuple may not be 32. * accounts/abi: add unit test of getTypeSize method

internal: fix personal.sign() (#21503)

ff23e26

go.mod : update goja dependency (#21432)

d90bbce

core, eth, trie: prepare trie sync for path based operation

eeaf191

Merge pull request #21504 from karalabe/trie-path-sync

f86324e

core, eth, trie: prepare trie sync for path based operation

accounts: abi/bid/backends; cleaned doc errors, camelCase refactors a…

c5d28f0

…nd anonymous variable assignments (#21514) Co-authored-by: Osoro Bironga <[email protected]>

accounts/usbwallet, signer/core: show accounts from ledger legacy der…

8327d1f

…ivation paths (#21517) * accounts/usbwallet, signer/core: un-hide accounts from ledger legacy derivation paths * Update accounts/usbwallet/wallet.go * Update signer/core/api.go * Update signer/core/api.go

build: remove wnode from the list of packages binaries (#21526)

066c755

.github: remove whisper from CODEOWNERS (#21527)

86bcbb0

params: update CHTs for v1.9.21 release

dc681fc

Merge pull request #21534 from karalabe/cht-1.9.21

24562d9

params: update CHTs for v1.9.21 release

params: release Geth v1.9.21

0287d54

Merge branch 'public_master' into upgrade/go-ethereum/v1.9.21-2021607…

4f45465

…125909

quorumbot added the geth upstream upgrade label Jun 9, 2021

ricardolyn assigned baptiste-b-pegasys Jun 9, 2021

Baptiste Boussemart and others added 3 commits June 9, 2021 06:31

fix: adapt new debug_traceCall method so it can handle private contra…

919c3a7

…cts without any signature change

Merge branch 'upgrade/go-ethereum/v1.9.21-2021607125909' into upgrade…

2dbc5dc

…/go-ethereum/v1.9.21-2021609142421

Merge branch 'master' into upgrade/go-ethereum/v1.9.21-2021609142421

2a7553e

Baptiste Boussemart and others added 2 commits June 14, 2021 08:31

revert new version of leveldb and protobuf

3f23db9

Merge branch 'master' into upgrade/go-ethereum/v1.9.21-2021609142421

d574312

ricardolyn marked this pull request as ready for review June 15, 2021 09:52

nmvalera approved these changes Jun 15, 2021

View reviewed changes

ricardolyn approved these changes Jun 15, 2021

View reviewed changes

ricardolyn merged commit 94d12a9 into Consensys:master Jun 15, 2021

ricardolyn deleted the upgrade/go-ethereum/v1.9.21-2021609142421 branch June 15, 2021 12:57

ricardolyn added this to the v21.4.X (Future Patch) milestone Jun 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Upgrade] Go-Ethereum release v1.9.21 #1211

[Upgrade] Go-Ethereum release v1.9.21 #1211

Uh oh!

quorumbot commented Jun 9, 2021 •

edited by ricardolyn

Loading

Uh oh!

nmvalera left a comment

Uh oh!

ricardolyn Jun 15, 2021

Uh oh!

baptiste-b-pegasys Jun 15, 2021 •

edited

Loading

Uh oh!

ricardolyn Jun 15, 2021

Uh oh!

ricardolyn Jun 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

[Upgrade] Go-Ethereum release v1.9.21 #1211

[Upgrade] Go-Ethereum release v1.9.21 #1211

Uh oh!

Conversation

quorumbot commented Jun 9, 2021 • edited by ricardolyn Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Plan & Analyse

Build & Test

Extra Changes & Tests

Left TODOs

Go-Ethereum Release: Tavum (v1.9.21)

Release notes

Codebase changes assessment

Legend

19 Pull Requests

72 Changed files

Uh oh!

nmvalera left a comment

Choose a reason for hiding this comment

Uh oh!

ricardolyn Jun 15, 2021

Choose a reason for hiding this comment

Uh oh!

baptiste-b-pegasys Jun 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ricardolyn Jun 15, 2021

Choose a reason for hiding this comment

Uh oh!

ricardolyn Jun 15, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

quorumbot commented Jun 9, 2021 •

edited by ricardolyn

Loading

baptiste-b-pegasys Jun 15, 2021 •

edited

Loading