Restart Loss Data

[Black-Memory]

I am using geth v1.10.25 and teku/v22.9.0
I don't know why geth was killed, there is no exit error, when I try to restart the service, geth starts to synchronize from around 15564000, why is so much data lost? Does anyone have the same problem as me?
And the synchronization speed is very slow, the total memory of my machine is 984G, and the available 234G

[rjl493456442]

Can you post the logs after restart here? I think your node is recovering by re-insert the lost blocks and should be recovered
a few minutes later.

[Black-Memory]

@rjl493456442
The lost data exceeds 4W blocks. It took about 5 hours to synchronize the data after restarting, and the disk frequently reads and writes data for processing.

[fernandezpaco]

Geth/v1.10.25-stable-69568c55/linux-amd64/go1.18.5
Prysm/v3.0.0

Same on my side, GETH dies, no crash error whatsoever, and when is restarted it goes back several days

My main concern is geth dying...

It happened to also on friday and it took 7hours to resync. Im running a extremely powerful machine.

today seems to resync faster

INFO [09-26|17:59:27.439] Imported new potential chain segment     blocks=1     txs=427        mgas=29.989   elapsed=3.188s       mgasps=9.404   number=15,619,184 hash=86bfc9..60ab0f dirty=4.39GiB
INFO [09-26|17:59:27.712] Chain head was updated                   number=15,619,184 hash=86bfc9..60ab0f root=650c20..66ef3f elapsed=9.892894ms
INFO [09-26|17:59:27.729] Unindexed transactions                   blocks=1     txs=22         tail=13,269,185 elapsed=16.476ms
INFO [09-26|17:59:36.025] Persisted the clean trie cache           path=/mnt/nvm/ether/geth/triecache          elapsed=13.134s
INFO [09-26|18:52:47.257] Starting Geth on Ethereum mainnet... 
INFO [09-26|18:52:47.262] Maximum peer count                       ETH=100 LES=0 total=100
INFO [09-26|18:52:47.263] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [09-26|18:52:47.267] Sanitizing cache to Go's GC limits       provided=24000 updated=20428
INFO [09-26|18:52:47.271] Set global gas cap                       cap=50,000,000
INFO [09-26|18:52:47.273] Allocated trie memory caches             clean=2.99GiB dirty=4.99GiB
INFO [09-26|18:52:47.275] Allocated cache and file handles         database=/mnt/nvm/ether/geth/chaindata cache=9.97GiB handles=32767
INFO [09-26|18:52:50.097] Found legacy ancient chain path          location=/mnt/nvm/ether/geth/chaindata/ancient
INFO [09-26|18:52:50.106] Opened ancient database                  database=/mnt/nvm/ether/geth/chaindata/ancient readonly=false
INFO [09-26|18:52:50.117]  
INFO [09-26|18:52:50.117] --------------------------------------------------------------------------------------------------------------------------------------------------------- 
INFO [09-26|18:52:50.117] Chain ID:  1 (mainnet) 
INFO [09-26|18:52:50.117] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work) 
INFO [09-26|18:52:50.117]  
INFO [09-26|18:52:50.117] Pre-Merge hard forks: 
INFO [09-26|18:52:50.117]  - Homestead:                   1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md) 
INFO [09-26|18:52:50.117]  - DAO Fork:                    1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md) 
INFO [09-26|18:52:50.117]  - Tangerine Whistle (EIP 150): 2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md) 
INFO [09-26|18:52:50.117]  - Spurious Dragon/1 (EIP 155): 2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) 
INFO [09-26|18:52:50.118]  - Spurious Dragon/2 (EIP 158): 2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md) 
INFO [09-26|18:52:50.118]  - Byzantium:                   4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md) 
INFO [09-26|18:52:50.118]  - Constantinople:              7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md) 
INFO [09-26|18:52:50.118]  - Petersburg:                  7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md) 
INFO [09-26|18:52:50.118]  - Istanbul:                    9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md) 
INFO [09-26|18:52:50.118]  - Muir Glacier:                9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md) 
INFO [09-26|18:52:50.118]  - Berlin:                      12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md) 
INFO [09-26|18:52:50.118]  - London:                      12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md) 
INFO [09-26|18:52:50.118]  - Arrow Glacier:               13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md) 
INFO [09-26|18:52:50.118]  - Gray Glacier:                15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md) 
INFO [09-26|18:52:50.118]  
INFO [09-26|18:52:50.118] Merge configured: 
INFO [09-26|18:52:50.118]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md 
INFO [09-26|18:52:50.118]  - Network known to be merged: true 
INFO [09-26|18:52:50.118]  - Total terminal difficulty:  58750000000000000000000 
INFO [09-26|18:52:50.118]  - Merge netsplit block:       <nil> 
INFO [09-26|18:52:50.118] --------------------------------------------------------------------------------------------------------------------------------------------------------- 
INFO [09-26|18:52:50.118]  
INFO [09-26|18:52:51.607] Disk storage enabled for ethash caches   dir=/mnt/nvm/ether/geth/ethash count=3
INFO [09-26|18:52:51.607] Disk storage enabled for ethash DAGs     dir=/home/ec2-user/.ethash     count=2
INFO [09-26|18:52:51.611] Initialising Ethereum protocol           network=1 dbversion=8
INFO [09-26|18:52:59.305] Loaded most recent local header          number=15,619,184 hash=86bfc9..60ab0f td=58,750,003,716,598,352,816,469 age=53m36s
INFO [09-26|18:52:59.306] Loaded most recent local full block      number=15,619,184 hash=86bfc9..60ab0f td=58,750,003,716,598,352,816,469 age=53m36s
INFO [09-26|18:52:59.306] Loaded most recent local fast block      number=15,619,184 hash=86bfc9..60ab0f td=58,750,003,716,598,352,816,469 age=53m36s
INFO [09-26|18:52:59.306] Loaded most recent local finalized block number=15,619,119 hash=6775dc..351f3c td=58,750,003,716,598,352,816,469 age=1h7m
INFO [09-26|18:52:59.308] Loaded last fast-sync pivot marker       number=14,795,858
WARN [09-26|18:52:59.310] Head state missing, repairing            number=15,619,184 hash=86bfc9..60ab0f snaproot=ab87f7..c5fbb7
INFO [09-26|18:54:17.301] Loaded most recent local header          number=15,619,184 hash=86bfc9..60ab0f td=58,750,003,716,598,352,816,469 age=54m54s
INFO [09-26|18:54:17.301] Loaded most recent local full block      number=15,595,089 hash=6db6a2..7f4887 td=58,750,003,716,598,352,816,469 age=3d9h38m
INFO [09-26|18:54:17.301] Loaded most recent local fast block      number=15,619,184 hash=86bfc9..60ab0f td=58,750,003,716,598,352,816,469 age=54m54s
INFO [09-26|18:54:17.301] Loaded most recent local finalized block number=15,619,119 hash=6775dc..351f3c td=58,750,003,716,598,352,816,469 age=1h8m18s
INFO [09-26|18:54:17.301] Loaded last fast-sync pivot marker       number=14,795,858
WARN [09-26|18:54:17.303] Enabling snapshot recovery               chainhead=15,595,089 diskbase=15,619,032
WARN [09-26|18:54:17.326] Loaded snapshot journal                  diskroot=ab87f7..c5fbb7 diffs=unmatched
WARN [09-26|18:54:17.326] Snapshot is not continuous with chain    snaproot=ab87f7..c5fbb7 chainroot=69eb85..931710
INFO [09-26|18:54:17.328] Loaded local transaction journal         transactions=2 dropped=0
INFO [09-26|18:54:17.328] Regenerated local transaction journal    transactions=2 accounts=2
WARN [09-26|18:54:17.328] Switch sync mode from snap sync to full sync 
INFO [09-26|18:54:17.328] Chain post-merge, sync via beacon client 
INFO [09-26|18:54:17.330] Gasprice oracle is ignoring threshold set threshold=2
WARN [09-26|18:54:17.330] Unclean shutdown detected                booted=2022-09-22T21:08:05+0000 age=3d21h46m
WARN [09-26|18:54:17.330] Unclean shutdown detected                booted=2022-09-26T17:59:22+0000 age=54m55s
WARN [09-26|18:54:17.330] Engine API enabled                       protocol=eth
INFO [09-26|18:54:17.332] Starting peer-to-peer node               instance=Geth/v1.10.25-stable-69568c55/linux-amd64/go1.18.5
INFO [09-26|18:54:17.450] HTTP server started                      endpoint=127.0.0.1:8545 auth=false prefix= cors= vhosts=localhost
INFO [09-26|18:54:17.450] WebSocket enabled                        url=ws://127.0.0.1:8546
INFO [09-26|18:54:17.451] WebSocket enabled                        url=ws://127.0.0.1:8551
INFO [09-26|18:54:17.451] HTTP server started                      endpoint=127.0.0.1:8551 auth=true  prefix= cors=localhost vhosts=localhost
WARN [09-26|18:54:26.008] State not available, ignoring new payload 
INFO [09-26|18:54:26.131] Forkchoice requested sync to new head    number=15,619,185 hash=b0eea8..c7adf6
INFO [09-26|18:54:26.467] Forkchoice requested sync to new head    number=15,619,186 hash=814d58..555066
INFO [09-26|18:54:26.746] Forkchoice requested sync to new head    number=15,619,187 hash=e0d3a5..74374a
INFO [09-26|18:54:26.977] Forkchoice requested sync to new head    number=15,619,188 hash=17bf90..7d445f
ERROR[09-26|18:54:27.144] Cleaning spurious beacon sync leftovers  stale_head_1=15,596,154 stale_tail_1=15,591,474 stale_next_1=3401d8..9e2142
INFO [09-26|18:54:27.144] Syncing beacon headers                   downloaded=1 left=0 eta=0s
INFO [09-26|18:54:27.144] Block synchronisation started 
WARN [09-26|18:54:27.147] Retrieved pivot header from local        number=15,619,124 hash=19b7d0..c3460f latest=15,619,188 oldest=15,619,184
INFO [09-26|18:54:27.282] Forkchoice requested sync to new head    number=15,619,189 hash=549196..4d0548
INFO [09-26|18:54:27.468] Looking for peers                        peercount=10 tried=15 static=0
INFO [09-26|18:54:27.477] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=5.55KiB throttle=8192
INFO [09-26|18:54:27.587] Forkchoice requested sync to new head    number=15,619,190 hash=2cabe8..d0f012
INFO [09-26|18:54:27.874] Forkchoice requested sync to new head    number=15,619,191 hash=4248d7..c58a9b
INFO [09-26|18:54:28.163] Forkchoice requested sync to new head    number=15,619,192 hash=e76377..99a7c1
INFO [09-26|18:54:28.442] Forkchoice requested sync to new head    number=15,619,193 hash=20b4a5..4ebac7
INFO [09-26|18:54:28.680] Forkchoice requested sync to new head    number=15,619,194 hash=0ad9c7..f53530
INFO [09-26|18:54:28.821] New local node record                    seq=1,652,809,142,480 id=714f45f05a771149 ip=34.238.243.60 udp=30303 tcp=30303
INFO [09-26|18:54:28.958] Forkchoice requested sync to new head    number=15,619,195 hash=ef663a..183252
INFO [09-26|18:54:29.223] Forkchoice requested sync to new head    number=15,619,196 hash=9f38fc..278f26
INFO [09-26|18:54:29.486] Forkchoice requested sync to new head    number=15,619,197 hash=99402c..7f3020
INFO [09-26|18:54:29.758] Forkchoice requested sync to new head    number=15,619,198 hash=c786c8..a9fc05
INFO [09-26|18:54:30.021] Forkchoice requested sync to new head    number=15,619,199 hash=6f7dc5..499267
INFO [09-26|18:54:30.335] Forkchoice requested sync to new head    number=15,619,200 hash=40ed1c..41b925
INFO [09-26|18:54:30.630] Forkchoice requested sync to new head    number=15,619,201 hash=d4a32f..6226c3
INFO [09-26|18:54:30.783] Importing heavy sidechain segment        blocks=856 start=15,595,090 end=15,595,945
INFO [09-26|18:54:31.299] Forkchoice requested sync to new head    number=15,619,202 hash=07a8a6..a44787
INFO [09-26|18:54:31.717] Forkchoice requested sync to new head    number=15,619,203 hash=015522..e1d2c1
INFO [09-26|18:54:32.225] Forkchoice requested sync to new head    number=15,619,204 hash=e9b701..88602e
INFO [09-26|18:54:32.597] Forkchoice requested sync to new head    number=15,619,205 hash=1309df..08448a
INFO [09-26|18:54:32.900] Forkchoice requested sync to new head    number=15,619,206 hash=4c1a7f..1d6a4b
INFO [09-26|18:54:33.230] Forkchoice requested sync to new head    number=15,619,207 hash=1fa858..c0cf92
INFO [09-26|18:54:33.602] Forkchoice requested sync to new head    number=15,619,208 hash=fcc3c4..7db2ab
INFO [09-26|18:54:33.962] Forkchoice requested sync to new head    number=15,619,209 hash=0cde1f..969900
INFO [09-26|18:54:34.385] Forkchoice requested sync to new head    number=15,619,210 hash=c53737..a7ee61
INFO [09-26|18:54:34.740] Forkchoice requested sync to new head    number=15,619,211 hash=e4bf29..e9c9d0
WARN [09-26|18:54:34.762] Snapshot extension registration failed   peer=971fdb34 err="peer connected on snap without compatible eth support"
INFO [09-26|18:54:35.063] Forkchoice requested sync to new head    number=15,619,212 hash=36ba23..f4efc2
INFO [09-26|18:54:37.506] Looking for peers                        peercount=17 tried=28 static=0
INFO [09-26|18:54:37.613] Forkchoice requested sync to new head    number=15,619,213 hash=e5b86b..a66433
INFO [09-26|18:54:38.146] Forkchoice requested sync to new head    number=15,619,214 hash=447601..b1b8c0
INFO [09-26|18:54:38.400] Forkchoice requested sync to new head    number=15,619,215 hash=ab797c..f6a18d
INFO [09-26|18:54:38.659] Forkchoice requested sync to new head    number=15,619,216 hash=6d3f5b..d10c8c
INFO [09-26|18:54:38.863] Forkchoice requested sync to new head    number=15,619,217 hash=07b13a..8d675a
INFO [09-26|18:54:39.084] Forkchoice requested sync to new head    number=15,619,218 hash=6c3d92..81b877
INFO [09-26|18:54:39.331] Forkchoice requested sync to new head    number=15,619,219 hash=e4759d..5c0b00
INFO [09-26|18:54:39.386] Imported new chain segment               blocks=5   txs=823 mgas=87.231 elapsed=8.602s mgasps=10.140 number=15,595,094 hash=ec5136..70a641 age=3d9h37m  dirty=7.74MiB
INFO [09-26|18:54:39.577] Forkchoice requested sync to new head    number=15,619,220 hash=79a223..761a4d
INFO [09-26|18:54:39.929] Forkchoice requested sync to new head    number=15,619,221 hash=8e4370..543e72
INFO [09-26|18:54:40.277] Forkchoice requested sync to new head    number=15,619,222 hash=906b73..277e8d
INFO [09-26|18:54:40.630] Forkchoice requested sync to new head    number=15,619,223 hash=8b411b..da968f
INFO [09-26|18:54:41.045] Forkchoice requested sync to new head    number=15,619,224 hash=0d8090..b5482b
INFO [09-26|18:54:41.501] Forkchoice requested sync to new head    number=15,619,225 hash=978db9..cc84ab
INFO [09-26|18:54:41.969] Forkchoice requested sync to new head    number=15,619,226 hash=eabf62..f4ca92
WARN [09-26|18:54:42.030] Checkpoint challenge timed out, dropping id=01be3621922eeb77 conn=inbound addr=16.170.220.52:60398 type=Geth/v1.1.9-5404d9d8...
INFO [09-26|18:54:42.383] Forkchoice requested sync to new head    number=15,619,227 hash=885aa3..df077c
INFO [09-26|18:54:42.776] Forkchoice requested sync to new head    number=15,619,228 hash=cc7448..134406
INFO [09-26|18:54:43.187] Forkchoice requested sync to new head    number=15,619,229 hash=375757..5e8c65
INFO [09-26|18:54:43.616] Forkchoice requested sync to new head    number=15,619,230 hash=154616..2b34c3
INFO [09-26|18:54:43.979] Forkchoice requested sync to new head    number=15,619,231 hash=df9903..7c8f8e
INFO [09-26|18:54:44.254] Forkchoice requested sync to new head    number=15,619,232 hash=61b131..28efcb
INFO [09-26|18:54:44.562] Forkchoice requested sync to new head    number=15,619,233 hash=a9340b..0e1738
INFO [09-26|18:54:44.885] Forkchoice requested sync to new head    number=15,619,234 hash=23bbb5..cc4907
INFO [09-26|18:54:45.198] Forkchoice requested sync to new head    number=15,619,235 hash=d307fb..9456c7
INFO [09-26|18:54:45.577] Forkchoice requested sync to new head    number=15,619,236 hash=ac5bcc..5bb52d
INFO [09-26|18:54:46.057] Forkchoice requested sync to new head    number=15,619,237 hash=bdcde5..376404
INFO [09-26|18:54:46.369] Forkchoice requested sync to new head    number=15,619,238 hash=ae17d9..23cb4b
INFO [09-26|18:54:46.637] Forkchoice requested sync to new head    number=15,619,239 hash=784021..d258f9
INFO [09-26|18:54:47.031] Forkchoice requested sync to new head    number=15,619,240 hash=8c1355..9827e0
INFO [09-26|18:54:47.516] Forkchoice requested sync to new head    number=15,619,241 hash=92e833..57b1c2
INFO [09-26|18:54:47.542] Looking for peers                        peercount=25 tried=34 static=0
INFO [09-26|18:54:47.787] Forkchoice requested sync to new head    number=15,619,242 hash=cafd80..9f73cf
INFO [09-26|18:54:48.018] Forkchoice requested sync to new head    number=15,619,243 hash=76bc33..aab789
INFO [09-26|18:54:48.307] Forkchoice requested sync to new head    number=15,619,244 hash=6075d3..7d570f
INFO [09-26|18:54:48.354] Imported new chain segment               blocks=7   txs=1524 mgas=113.328 elapsed=8.968s mgasps=12.637 number=15,595,101 hash=d6d88d..3e51b2 age=3d9h36m  dirty=18.98MiB
INFO [09-26|18:54:49.630] Forkchoice requested sync to new head    number=15,619,245 hash=d13669..680706
INFO [09-26|18:54:50.124] Forkchoice requested sync to new head    number=15,619,246 hash=4e4898..9cef50
INFO [09-26|18:54:50.390] Forkchoice requested sync to new head    number=15,619,247 hash=9d4c92..0d84fc
INFO [09-26|18:54:50.646] Forkchoice requested sync to new head    number=15,619,248 hash=df33d9..63cc0e
INFO [09-26|18:54:50.891] Forkchoice requested sync to new head    number=15,619,249 hash=376983..881766
INFO [09-26|18:54:51.163] Forkchoice requested sync to new head    number=15,619,250 hash=791918..2ade85
INFO [09-26|18:54:51.407] Forkchoice requested sync to new head    number=15,619,251 hash=a24670..a98178
INFO [09-26|18:54:51.672] Forkchoice requested sync to new head    number=15,619,252 hash=36439d..962aaf
INFO [09-26|18:54:51.943] Forkchoice requested sync to new head    number=15,619,253 hash=cf7c19..2e5510
INFO [09-26|18:54:52.201] Forkchoice requested sync to new head    number=15,619,254 hash=9a3d98..8e391f
INFO [09-26|18:54:52.419] Forkchoice requested sync to new head    number=15,619,255 hash=c4afa5..a992ce
INFO [09-26|18:54:52.666] Forkchoice requested sync to new head    number=15,619,256 hash=9a7455..f68821
INFO [09-26|18:54:52.886] Forkchoice requested sync to new head    number=15,619,257 hash=b3e9cc..995f53
INFO [09-26|18:54:53.113] Forkchoice requested sync to new head    number=15,619,258 hash=e4ba34..1a575b
INFO [09-26|18:54:53.392] Forkchoice requested sync to new head    number=15,619,259 hash=955191..c098fb
INFO [09-26|18:54:53.607] Forkchoice requested sync to new head    number=15,619,260 hash=705cb2..6478ac
INFO [09-26|18:54:53.869] Forkchoice requested sync to new head    number=15,619,261 hash=56fe0f..8a30f1
WARN [09-26|18:54:54.112] Snapshot extension registration failed   peer=3422d49f err="peer connected on snap without compatible eth support"
INFO [09-26|18:54:54.152] Forkchoice requested sync to new head    number=15,619,262 hash=81f249..5038cb
INFO [09-26|18:54:54.383] Forkchoice requested sync to new head    number=15,619,263 hash=645030..c454f5
INFO [09-26|18:54:54.647] Forkchoice requested sync to new head    number=15,619,264 hash=555701..d4b33c
INFO [09-26|18:54:54.846] Forkchoice requested sync to new head    number=15,619,265 hash=788159..eaa173
INFO [09-26|18:54:55.116] Forkchoice requested sync to new head    number=15,619,266 hash=613896..4b8d6d
INFO [09-26|18:54:55.381] Forkchoice requested sync to new head    number=15,619,267 hash=18b959..a8f171
INFO [09-26|18:54:55.705] Forkchoice requested sync to new head    number=15,619,268 hash=646e40..18aa38
INFO [09-26|18:54:55.933] Forkchoice requested sync to new head    number=15,619,269 hash=200079..a78807
INFO [09-26|18:54:56.188] Forkchoice requested sync to new head    number=15,619,270 hash=6a7d95..c9f9f8
INFO [09-26|18:54:56.479] Forkchoice requested sync to new head    number=15,619,271 hash=cf7e95..5311e9
INFO [09-26|18:54:56.740] Forkchoice requested sync to new head    number=15,619,272 hash=6062e5..c17c9d
INFO [09-26|18:54:57.014] Forkchoice requested sync to new head    number=15,619,273 hash=e65d78..18895c
INFO [09-26|18:54:57.227] Imported new chain segment               blocks=10  txs=1408 mgas=144.006 elapsed=8.872s mgasps=16.231 number=15,595,111 hash=5e4089..6e65fe age=3d9h34m  dirty=34.82MiB
INFO [09-26|18:54:57.258] Forkchoice requested sync to new head    number=15,619,274 hash=152234..df176e
INFO [09-26|18:54:57.505] Forkchoice requested sync to new head    number=15,619,275 hash=b5f6f3..3fb381
INFO [09-26|18:54:57.747] Forkchoice requested sync to new head    number=15,619,276 hash=12ec0d..1b277f
INFO [09-26|18:54:57.753] Looking for peers                        peercount=28 tried=29 static=0
INFO [09-26|18:54:58.764] Forkchoice requested sync to new head    number=15,619,277 hash=8ba548..b934de
INFO [09-26|18:54:59.257] Forkchoice requested sync to new head    number=15,619,278 hash=cefa83..524843
INFO [09-26|18:54:59.519] Forkchoice requested sync to new head    number=15,619,279 hash=f4f48d..baff32
INFO [09-26|18:54:59.752] Forkchoice requested sync to new head    number=15,619,280 hash=6db81f..136da1
INFO [09-26|18:54:59.973] Forkchoice requested sync to new head    number=15,619,281 hash=6b64bb..80110c
INFO [09-26|18:55:00.267] Forkchoice requested sync to new head    number=15,619,282 hash=636eac..e805f9
INFO [09-26|18:55:00.505] Forkchoice requested sync to new head    number=15,619,283 hash=e991e0..905e06
INFO [09-26|18:55:00.776] Forkchoice requested sync to new head    number=15,619,284 hash=bae289..463bb0
INFO [09-26|18:55:01.048] Forkchoice requested sync to new head    number=15,619,285 hash=d6588c..916dfc
INFO [09-26|18:55:01.325] Forkchoice requested sync to new head    number=15,619,286 hash=c25ed7..ba806e
INFO [09-26|18:55:01.570] Forkchoice requested sync to new head    number=15,619,287 hash=acce30..1869f3
INFO [09-26|18:55:01.831] Forkchoice requested sync to new head    number=15,619,288 hash=49ee7e..36332e
INFO [09-26|18:55:02.086] Forkchoice requested sync to new head    number=15,619,289 hash=68ac18..5b1e37
INFO [09-26|18:55:02.326] Forkchoice requested sync to new head    number=15,619,290 hash=a104db..9458d1
INFO [09-26|18:55:02.575] Forkchoice requested sync to new head    number=15,619,291 hash=1126aa..f3b058
INFO [09-26|18:55:02.806] Forkchoice requested sync to new head    number=15,619,292 hash=7dcea4..57dfd5
INFO [09-26|18:55:03.066] Forkchoice requested sync to new head    number=15,619,293 hash=1fd357..f4a755
INFO [09-26|18:55:03.302] Forkchoice requested sync to new head    number=15,619,294 hash=1ea940..5c0417
INFO [09-26|18:55:03.545] Forkchoice requested sync to new head    number=15,619,295 hash=9927cc..a3d6ff
INFO [09-26|18:55:03.821] Forkchoice requested sync to new head    number=15,619,296 hash=02bc28..ad9e5c
INFO [09-26|18:55:04.044] Forkchoice requested sync to new head    number=15,619,297 hash=4cfe9e..16d95d
INFO [09-26|18:55:04.328] Forkchoice requested sync to new head    number=15,619,298 hash=e3e2c5..191e72
INFO [09-26|18:55:04.558] Forkchoice requested sync to new head    number=15,619,299 hash=8eec08..5327ab
INFO [09-26|18:55:04.897] Forkchoice requested sync to new head    number=15,619,300 hash=222723..ad7c81
INFO [09-26|18:55:05.115] Forkchoice requested sync to new head    number=15,619,301 hash=009c23..87e5ec
INFO [09-26|18:55:05.381] Forkchoice requested sync to new head    number=15,619,302 hash=dcd0ed..911e1c
INFO [09-26|18:55:05.626] Forkchoice requested sync to new head    number=15,619,303 hash=215a80..a04465
INFO [09-26|18:55:05.882] Forkchoice requested sync to new head    number=15,619,304 hash=060959..e18f79
INFO [09-26|18:55:06.156] Forkchoice requested sync to new head    number=15,619,305 hash=79699c..9b610f
INFO [09-26|18:55:06.400] Forkchoice requested sync to new head    number=15,619,306 hash=7578d0..077f3e
INFO [09-26|18:55:06.432] Imported new chain segment               blocks=12  txs=1471 mgas=163.875 elapsed=9.205s mgasps=17.802 number=15,595,123 hash=c834f9..a9e4ff age=3d9h32m  dirty=52.71MiB
INFO [09-26|18:55:06.677] Forkchoice requested sync to new head    number=15,619,307 hash=5ad229..061ab4
INFO [09-26|18:55:06.976] Forkchoice requested sync to new head    number=15,619,308 hash=374c26..9deedb
INFO [09-26|18:55:07.960] Looking for peers                        peercount=31 tried=29 static=0
INFO [09-26|18:55:08.276] Forkchoice requested sync to new head    number=15,619,309 hash=96607f..2c8860
INFO [09-26|18:55:08.824] Forkchoice requested sync to new head    number=15,619,310 hash=d328c7..afe423
INFO [09-26|18:55:09.313] Forkchoice requested sync to new head    number=15,619,311 hash=50dfb4..1ceb6a
INFO [09-26|18:55:09.603] Forkchoice requested sync to new head    number=15,619,312 hash=425fbb..3d2ecb
INFO [09-26|18:55:10.404] Forkchoice requested sync to new head    number=15,619,313 hash=52c8f9..22880e
INFO [09-26|18:55:10.665] Forkchoice requested sync to new head    number=15,619,314 hash=5fc8c8..9189c6
INFO [09-26|18:55:10.954] Forkchoice requested sync to new head    number=15,619,315 hash=f16b0d..8fe136
INFO [09-26|18:55:11.232] Forkchoice requested sync to new head    number=15,619,316 hash=6c23e9..c813fd
INFO [09-26|18:55:11.468] Forkchoice requested sync to new head    number=15,619,317 hash=00ba8f..e5fd22
INFO [09-26|18:55:11.719] Forkchoice requested sync to new head    number=15,619,318 hash=8c5174..eae13f
INFO [09-26|18:55:11.942] Forkchoice requested sync to new head    number=15,619,319 hash=fb1122..566167
INFO [09-26|18:55:12.201] Forkchoice requested sync to new head    number=15,619,320 hash=76e09b..6c3322
INFO [09-26|18:55:12.430] Forkchoice requested sync to new head    number=15,619,321 hash=ca945b..4db210
INFO [09-26|18:55:12.668] Forkchoice requested sync to new head    number=15,619,322 hash=6bc6ef..aa912c
INFO [09-26|18:55:13.020] Forkchoice requested sync to new head    number=15,619,323 hash=787000..61a6a7
INFO [09-26|18:55:13.245] Forkchoice requested sync to new head    number=15,619,324 hash=6be231..06bcba
INFO [09-26|18:55:13.506] Forkchoice requested sync to new head    number=15,619,325 hash=fd34ac..fcc7c5
INFO [09-26|18:55:13.717] Forkchoice requested sync to new head    number=15,619,326 hash=177dd2..0b0d0a
INFO [09-26|18:55:14.007] Forkchoice requested sync to new head    number=15,619,327 hash=ad70b7..3ed2fb
INFO [09-26|18:55:14.215] Forkchoice requested sync to new head    number=15,619,328 hash=216604..3407f0
INFO [09-26|18:55:14.437] Forkchoice requested sync to new head    number=15,619,329 hash=0b507a..4e6107
INFO [09-26|18:55:14.445] Imported new chain segment               blocks=12  txs=1520 mgas=177.734 elapsed=8.012s mgasps=22.182 number=15,595,135 hash=69acf3..0e1113 age=3d9h30m  dirty=70.89MiB
INFO [09-26|18:55:14.655] Forkchoice requested sync to new head    number=15,619,330 hash=fa4459..98b52f
INFO [09-26|18:55:14.874] Forkchoice requested sync to new head    number=15,619,331 hash=e32e77..abf6ea
INFO [09-26|18:55:15.192] Forkchoice requested sync to new head    number=15,619,332 hash=4c5ba4..ceb8bc
INFO [09-26|18:55:15.406] Forkchoice requested sync to new head    number=15,619,333 hash=72aff6..f4b146
INFO [09-26|18:55:15.649] Forkchoice requested sync to new head    number=15,619,334 hash=ef6b15..f64a7d
INFO [09-26|18:55:15.858] Forkchoice requested sync to new head    number=15,619,335 hash=7ea389..0494ea
INFO [09-26|18:55:16.077] Forkchoice requested sync to new head    number=15,619,336 hash=323255..c886b3
INFO [09-26|18:55:16.293] Forkchoice requested sync to new head    number=15,619,337 hash=a22026..ff38d0
INFO [09-26|18:55:16.536] Forkchoice requested sync to new head    number=15,619,338 hash=5c3763..df6a94
INFO [09-26|18:55:16.753] Forkchoice requested sync to new head    number=15,619,339 hash=b705b9..922a26
INFO [09-26|18:55:17.017] Forkchoice requested sync to new head    number=15,619,340 hash=c8da9d..7a0808
INFO [09-26|18:55:17.610] Forkchoice requested sync to new head    number=15,619,341 hash=d89fdd..19a0f4
INFO [09-26|18:55:18.086] Looking for peers                        peercount=38 tried=39 static=0
INFO [09-26|18:55:18.483] Forkchoice requested sync to new head    number=15,619,342 hash=a45536..760d22
INFO [09-26|18:55:18.722] Forkchoice requested sync to new head    number=15,619,343 hash=4ca0e4..baab90
INFO [09-26|18:55:19.008] Forkchoice requested sync to new head    number=15,619,344 hash=6464ca..001f60
INFO [09-26|18:55:19.252] Forkchoice requested sync to new head    number=15,619,345 hash=fe4742..259709
INFO [09-26|18:55:19.501] Forkchoice requested sync to new head    number=15,619,346 hash=0c93ce..6488df
INFO [09-26|18:55:19.740] Forkchoice requested sync to new head    number=15,619,347 hash=2981d3..a458ec
INFO [09-26|18:55:20.025] Forkchoice requested sync to new head    number=15,619,348 hash=b2ab34..fde264
INFO [09-26|18:55:20.290] Forkchoice requested sync to new head    number=15,619,349 hash=5769bc..16c49a
INFO [09-26|18:55:20.579] Forkchoice requested sync to new head    number=15,619,350 hash=e9d8db..ced54a
INFO [09-26|18:55:20.853] Forkchoice requested sync to new head    number=15,619,351 hash=8ad475..368f86
INFO [09-26|18:55:21.149] Forkchoice requested sync to new head    number=15,619,352 hash=227463..cc04ea
INFO [09-26|18:55:21.387] Forkchoice requested sync to new head    number=15,619,353 hash=59ec9b..075555
INFO [09-26|18:55:21.626] Forkchoice requested sync to new head    number=15,619,354 hash=24d58c..90017b
INFO [09-26|18:55:21.899] Forkchoice requested sync to new head    number=15,619,355 hash=823051..ef4179
INFO [09-26|18:55:22.215] Forkchoice requested sync to new head    number=15,619,356 hash=7ec417..d9102c
INFO [09-26|18:55:22.555] Forkchoice requested sync to new head    number=15,619,357 hash=b312b3..650399
INFO [09-26|18:55:22.818] Forkchoice requested sync to new head    number=15,619,358 hash=fc8528..1e71eb
INFO [09-26|18:55:23.127] Forkchoice requested sync to new head    number=15,619,359 hash=e3dea6..8787b1
INFO [09-26|18:55:23.393] Forkchoice requested sync to new head    number=15,619,360 hash=53a9ff..0b6dcd
INFO [09-26|18:55:23.624] Forkchoice requested sync to new head    number=15,619,361 hash=31ab37..2d3320
INFO [09-26|18:55:23.825] Forkchoice requested sync to new head    number=15,619,362 hash=b48047..bf1235
INFO [09-26|18:55:23.921] Imported new chain segment               blocks=12  txs=2032 mgas=194.808 elapsed=9.476s mgasps=20.557 number=15,595,147 hash=269ddd..d06e8c age=3d9h28m  dirty=92.04MiB
INFO [09-26|18:55:24.111] Forkchoice requested sync to new head    number=15,619,363 hash=d81b75..843066
INFO [09-26|18:55:24.463] Forkchoice requested sync to new head    number=15,619,364 hash=3cdc15..575037
INFO [09-26|18:55:24.696] Forkchoice requested sync to new head    number=15,619,365 hash=67751e..1e1466
INFO [09-26|18:55:24.944] Forkchoice requested sync to new head    number=15,619,366 hash=5a56a9..ce6c15
INFO [09-26|18:55:25.213] Forkchoice requested sync to new head    number=15,619,367 hash=a16d82..1c3e92
INFO [09-26|18:55:25.488] Forkchoice requested sync to new head    number=15,619,368 hash=dfba54..85de8b
INFO [09-26|18:55:25.841] Forkchoice requested sync to new head    number=15,619,369 hash=397064..df92fc
INFO [09-26|18:55:26.130] Forkchoice requested sync to new head    number=15,619,370 hash=22bad4..d2f4bb
INFO [09-26|18:55:26.453] Forkchoice requested sync to new head    number=15,619,371 hash=4a1775..29ec78
INFO [09-26|18:55:27.854] Forkchoice requested sync to new head    number=15,619,372 hash=c59507..7e89a4
INFO [09-26|18:55:28.087] Looking for peers                        peercount=44 tried=30 static=0
INFO [09-26|18:55:28.694] Forkchoice requested sync to new head    number=15,619,373 hash=f9fb87..72abf7
INFO [09-26|18:55:28.958] Forkchoice requested sync to new head    number=15,619,374 hash=a0bdc2..caf8c4
INFO [09-26|18:55:29.310] Forkchoice requested sync to new head    number=15,619,375 hash=7176e8..120f1d
INFO [09-26|18:55:29.557] Forkchoice requested sync to new head    number=15,619,376 hash=1baa93..1a721a
INFO [09-26|18:55:29.863] Forkchoice requested sync to new head    number=15,619,377 hash=e74b18..5715e3
INFO [09-26|18:55:30.103] Forkchoice requested sync to new head    number=15,619,378 hash=2dfec3..107b7d
INFO [09-26|18:55:30.335] Forkchoice requested sync to new head    number=15,619,379 hash=39e7c4..a756e5
INFO [09-26|18:55:30.694] Forkchoice requested sync to new head    number=15,619,380 hash=e20979..8c5630
INFO [09-26|18:55:30.941] Forkchoice requested sync to new head    number=15,619,381 hash=56ce2a..ebf3fb
INFO [09-26|18:55:31.211] Forkchoice requested sync to new head    number=15,619,382 hash=6c0518..60bf26
INFO [09-26|18:55:31.481] Forkchoice requested sync to new head    number=15,619,383 hash=b15008..e8a057
INFO [09-26|18:55:31.774] Forkchoice requested sync to new head    number=15,619,384 hash=f013ef..6a5eb5
INFO [09-26|18:55:32.084] Forkchoice requested sync to new head    number=15,619,385 hash=abb77f..8914ff
INFO [09-26|18:55:32.330] Forkchoice requested sync to new head    number=15,619,386 hash=aeb883..be6493
INFO [09-26|18:55:32.649] Forkchoice requested sync to new head    number=15,619,387 hash=8d821f..6ffcb6
INFO [09-26|18:55:32.780] Imported new chain segment               blocks=14  txs=1938 mgas=213.168 elapsed=8.859s mgasps=24.061 number=15,595,161 hash=6557b6..b926c2 age=3d9h25m  dirty=114.27MiB
WARN [09-26|18:55:32.781] Inserted known block                     number=15,595,162 hash=fe00d1..4e9170 uncles=0 txs=0    gas=0 root=eafc41..df6637
INFO [09-26|18:55:32.934] Forkchoice requested sync to new head    number=15,619,388 hash=6cd16c..564dc9
INFO [09-26|18:55:33.210] Forkchoice requested sync to new head    number=15,619,389 hash=159568..373757
INFO [09-26|18:55:33.531] Forkchoice requested sync to new head    number=15,619,390 hash=a4fc11..9654aa
INFO [09-26|18:55:33.770] Forkchoice requested sync to new head    number=15,619,391 hash=320d10..6a7906
INFO [09-26|18:55:34.084] Forkchoice requested sync to new head    number=15,619,392 hash=f47097..4993b8
INFO [09-26|18:55:34.308] Forkchoice requested sync to new head    number=15,619,393 hash=25f804..fdfa75
INFO [09-26|18:55:34.550] Forkchoice requested sync to new head    number=15,619,394 hash=22f1a3..09f69a
INFO [09-26|18:55:34.910] Forkchoice requested sync to new head    number=15,619,395 hash=c8689b..2fb6f4
INFO [09-26|18:55:35.224] Forkchoice requested sync to new head    number=15,619,396 hash=f6a213..a46022
INFO [09-26|18:55:35.468] Forkchoice requested sync to new head    number=15,619,397 hash=96fdba..ee0b76
INFO [09-26|18:55:35.751] Forkchoice requested sync to new head    number=15,619,398 hash=d9db85..630192
INFO [09-26|18:55:36.018] Forkchoice requested sync to new head    number=15,619,399 hash=6ee14c..3fbcf2
INFO [09-26|18:55:36.314] Forkchoice requested sync to new head    number=15,619,400 hash=086c70..5f2331
INFO [09-26|18:55:36.611] Forkchoice requested sync to new head    number=15,619,401 hash=c01f04..af6f65
INFO [09-26|18:55:36.946] Forkchoice requested sync to new head    number=15,619,402 hash=eacbd3..7b9564
INFO [09-26|18:55:37.223] Forkchoice requested sync to new head    number=15,619,403 hash=a0f9e2..c10207
INFO [09-26|18:55:38.091] Looking for peers                        peercount=48 tried=39 static=0
INFO [09-26|18:55:38.446] Forkchoice requested sync to new head    number=15,619,404 hash=c00ea9..fe0b86
INFO [09-26|18:55:39.095] Forkchoice requested sync to new head    number=15,619,405 hash=c72c81..de5b14
INFO [09-26|18:55:39.324] Forkchoice requested sync to new head    number=15,619,406 hash=89aa28..ed2a28
INFO [09-26|18:55:39.575] Forkchoice requested sync to new head    number=15,619,407 hash=59b2bf..db8b76
INFO [09-26|18:55:39.832] Forkchoice requested sync to new head    number=15,619,408 hash=f30d36..1d1981
INFO [09-26|18:55:40.066] Forkchoice requested sync to new head    number=15,619,409 hash=4a5684..6f486c
INFO [09-26|18:55:40.314] Forkchoice requested sync to new head    number=15,619,410 hash=4b7690..35b3a4
INFO [09-26|18:55:40.525] Forkchoice requested sync to new head    number=15,619,411 hash=151ffd..bd241a
INFO [09-26|18:55:40.752] Forkchoice requested sync to new head    number=15,619,412 hash=df4ed5..57f460
INFO [09-26|18:55:40.868] Imported new chain segment               blocks=15  txs=1938 mgas=242.469 elapsed=8.087s mgasps=29.982 number=15,595,176 hash=b67da6..f417e6 age=3d9h22m  dirty=135.95MiB
IN

[fernandezpaco]

Has just finished resynchronizing. It took 1hour more or less.

When the resync has reached the point where geth crashed i got this logs:

ERROR[09-26|20:10:00.502] Unhandled trie error: missing trie node dfd3efbcf32b13b52e5a98222bef237e18d5b8feb92d546ec4c218cacd33570d (path 0705) <nil> 
ERROR[09-26|20:10:00.502] Unhandled trie error: missing trie node dfd3efbcf32b13b52e5a98222bef237e18d5b8feb92d546ec4c218cacd33570d (path 0705) <nil> 
ERROR[09-26|20:10:00.504] Unhandled trie error: missing trie node 4feb10ddf30d15965a6d2b79010f25f62e634db9a6ca8d1d469ea2dcab99070b (path 0208) <nil> 
ERROR[09-26|20:10:00.504] Unhandled trie error: missing trie node 4feb10ddf30d15965a6d2b79010f25f62e634db9a6ca8d1d469ea2dcab99070b (path 0208) <nil> 
ERROR[09-26|20:10:00.506] Unhandled trie error: missing trie node bf10297cf08eeef536a452d18daf525ebb1ac35020995e4baca10fe62bc950cb (path 0b06) <nil> 
ERROR[09-26|20:10:00.506] Unhandled trie error: missing trie node bf10297cf08eeef536a452d18daf525ebb1ac35020995e4baca10fe62bc950cb (path 0b06) <nil> 
ERROR[09-26|20:10:00.507] Unhandled trie error: missing trie node 6101f36900a8f717ed65bd36857ed7fcef37e0a6746ae1604d6890d40185ee0c (path 0803) <nil> 
ERROR[09-26|20:10:00.507] Unhandled trie error: missing trie node 6101f36900a8f717ed65bd36857ed7fcef37e0a6746ae1604d6890d40185ee0c (path 0803) <nil> 
ERROR[09-26|20:10:00.509] Unhandled trie error: missing trie node 13bda017b8e6d70c3a244fee3501251f588050485d37f8057cd8952621d19d29 (path 0b0f) <nil> 
ERROR[09-26|20:10:00.509] Unhandled trie error: missing trie node 13bda017b8e6d70c3a244fee3501251f588050485d37f8057cd8952621d19d29 (path 0b0f) <nil> 
ERROR[09-26|20:10:00.511] Unhandled trie error: missing trie node 5887b37b5897258ff092ca6cf764eb704521f5d116150d5d6131fcbcde75d386 (path 030d) <nil> 
ERROR[09-26|20:10:00.511] Unhandled trie error: missing trie node 5887b37b5897258ff092ca6cf764eb704521f5d116150d5d6131fcbcde75d386 (path 030d) <nil> 
ERROR[09-26|20:10:00.546] Unhandled trie error: missing trie node f4e5dbafbe9061c327a396661bd46569ff09c26264f390d32bb6ae303331cfef (path 020b) <nil> 
ERROR[09-26|20:10:00.546] Unhandled trie error: missing trie node f4e5dbafbe9061c327a396661bd46569ff09c26264f390d32bb6ae303331cfef (path 020b) <nil> 
ERROR[09-26|20:10:00.573] Unhandled trie error: missing trie node 59da14992b5e431edb9e377ca4fca1d6c76caa21e80e4023a5eb5c628b9f44b1 (path 0601) <nil> 
ERROR[09-26|20:10:00.573] Unhandled trie error: missing trie node 59da14992b5e431edb9e377ca4fca1d6c76caa21e80e4023a5eb5c628b9f44b1 (path 0601) <nil> 
ERROR[09-26|20:10:00.573] Unhandled trie error: missing trie node e0e559a6485282a6478038287b9d2c59eabd9f35f0009a8b251a9f5747ff7528 (path 0108) <nil> 
ERROR[09-26|20:10:00.574] Unhandled trie error: missing trie node e0e559a6485282a6478038287b9d2c59eabd9f35f0009a8b251a9f5747ff7528 (path 0108) <nil> 
ERROR[09-26|20:10:00.574] Unhandled trie error: missing trie node f8e6cbc65fc5e1a55462ae041ab9be5adf55cb9b174c153fd708775825dfb520 (path 0f) <nil> 
ERROR[09-26|20:10:00.574] Unhandled trie error: missing trie node f8e6cbc65fc5e1a55462ae041ab9be5adf55cb9b174c153fd708775825dfb520 (path 0f) <nil> 
ERROR[09-26|20:10:00.574] Unhandled trie error: missing trie node f8e6cbc65fc5e1a55462ae041ab9be5adf55cb9b174c153fd708775825dfb520 (path 0f) <nil> 
ERROR[09-26|20:10:00.574] Unhandled trie error: missing trie node f8e6cbc65fc5e1a55462ae041ab9be5adf55cb9b174c153fd708775825dfb520 (path 0f) <nil> 
ERROR[09-26|20:10:00.574] Unhandled trie error: missing trie node 8f349ecb0ec8a455161bb1e7c3f61a76ff55942394fd93ead1c10eb11cef9c27 (path 0901) <nil> 
ERROR[09-26|20:10:00.574] Unhandled trie error: missing trie node 8f349ecb0ec8a455161bb1e7c3f61a76ff55942394fd93ead1c10eb11cef9c27 (path 0901) <nil> 
ERROR[09-26|20:10:00.574] Unhandled trie error: missing trie node 8f349ecb0ec8a455161bb1e7c3f61a76ff55942394fd93ead1c10eb11cef9c27 (path 0901) <nil> 
ERROR[09-26|20:10:00.575] Unhandled trie error: missing trie node 8f349ecb0ec8a455161bb1e7c3f61a76ff55942394fd93ead1c10eb11cef9c27 (path 0901) <nil> 
ERROR[09-26|20:10:00.575] Unhandled trie error: missing trie node 2e5ea817743cb7997269f0141bdec476c84eaaeabf72dccc010956138230a9ff (path 060c) <nil> 
ERROR[09-26|20:10:00.575] Unhandled trie error: missing trie node 2e5ea817743cb7997269f0141bdec476c84eaaeabf72dccc010956138230a9ff (path 060c) <nil> 
ERROR[09-26|20:10:00.575] Unhandled trie error: missing trie node 020aa4ef47b73032539a49db31bdd1131114b5d203de4eeebb3272bb54c6bae8 (path 0b03) <nil> 
ERROR[09-26|20:10:00.575] Unhandled trie error: missing trie node 020aa4ef47b73032539a49db31bdd1131114b5d203de4eeebb3272bb54c6bae8 (path 0b03) <nil> 
ERROR[09-26|20:10:00.575] Unhandled trie error: missing trie node d2aad1b194be3a449a820fb8f2bededdbc0552993fc695d13cfd69272118d26c (path 0e05) <nil> 


But finally it finished ok:
RROR[09-26|20:10:21.892] Unhandled trie error: missing trie node e5cb5a9769aee5e53fd7b31f8d3b86629011e4610724fc58f71c9fe49354b37d (path 0805) <nil> 
ERROR[09-26|20:10:21.892] Unhandled trie error: missing trie node e5cb5a9769aee5e53fd7b31f8d3b86629011e4610724fc58f71c9fe49354b37d (path 0805) <nil> 
ERROR[09-26|20:10:21.913] Unhandled trie error: missing trie node 00e0c7ca6da1cb48c0e66c123aa13365adf1c8066151ceb16c0e59af4dca8576 (path 0800) <nil> 
ERROR[09-26|20:10:21.913] Unhandled trie error: missing trie node 00e0c7ca6da1cb48c0e66c123aa13365adf1c8066151ceb16c0e59af4dca8576 (path 0800) <nil> 
ERROR[09-26|20:10:21.920] Unhandled trie error: missing trie node ff3ff5b765c24b8ecdc384b72e4510bf03d0a35846d2e62ba25dde1708a4d9da (path 050c) <nil> 
ERROR[09-26|20:10:21.920] Unhandled trie error: missing trie node ff3ff5b765c24b8ecdc384b72e4510bf03d0a35846d2e62ba25dde1708a4d9da (path 050c) <nil> 
INFO [09-26|20:10:21.921] Imported new chain segment               blocks=19    txs=3100      mgas=299.135  elapsed=3.620s    mgasps=82.612  number=15,619,820 hash=b61bf9..b2d981 age=2m34s     dirty=4.47GiB
INFO [09-26|20:10:21.922] Downloader queue stats                   receiptTasks=0 blockTasks=0 itemSize=65.53KiB throttle=4001
INFO [09-26|20:10:23.327] Unindexed transactions                   blocks=535   txs=101,897   tail=13,269,821 elapsed=1.405s
INFO [09-26|20:10:24.106] Forkchoice requested sync to new head    number=15,619,833 hash=c4c501..d3199d
INFO [09-26|20:10:26.578] Imported new chain segment               blocks=13    txs=2125      mgas=207.993  elapsed=2.490s    mgasps=83.526  number=15,619,833 hash=c4c501..d3199d dirty=4.47GiB    ignored=1
INFO [09-26|20:10:26.598] Unindexed transactions                   blocks=13    txs=2554      tail=13,269,834 elapsed=20.135ms
INFO [09-26|20:10:36.748] Imported new potential chain segment     blocks=1     txs=235       mgas=19.221   elapsed=155.847ms mgasps=123.329 number=15,619,834 hash=9a6590..753f2e dirty=4.47GiB
INFO [09-26|20:10:36.811] Chain head was updated                   number=15,619,834 hash=9a6590..753f2e root=933e5c..0d39fd elapsed=8.437419ms
INFO [09-26|20:10:36.816] Unindexed transactions                   blocks=1     txs=290       tail=13,269,835 elapsed=4.508ms

[rjl493456442]

@fernandezpaco Unhandled trie error: missing trie node It's not a big issue and we have hidden this log in master branch. See #25640 for details.

[rjl493456442]

Geth has to recover the lost block by re-importing them. And it can take some time depends on how many blocks have been lost. Close the ticket since everything is expected.

[fernandezpaco]

Well geth crashing two times in four days is not as expected i guess...

[rjl493456442]

@fernandezpaco any crash is definitely unexpected. Would be appreciate you can open another ticket with the crash logs so that we can investigate a bit.

[Sajjon]

@fernandezpaco I just upgraded to v1.11.5 and unfortunately did an unclean exit of geth, now when I run I got:

Cleaning spurious beacon sync leftovers

Which you got to, do I have to sync from scratch now :/ ?

more log lines around it:

WARN [04-03|21:04:34.844] Ignoring payload while snap syncing      number=16,970,410 hash=32d99f..cf698a
INFO [04-03|21:04:35.298] Forkchoice requested sync to new head    number=16,970,407 hash=76ee67..e377ff finalized=unknown
WARN [04-03|21:04:35.364] Ignoring payload while snap syncing      number=16,970,411 hash=2745f1..cfc9d4
WARN [04-03|21:04:35.461] Ignoring payload while snap syncing      number=16,970,412 hash=fec72f..8b66e3
WARN [04-03|21:04:35.541] Ignoring payload while snap syncing      number=16,970,413 hash=46f1c0..dd16b4
INFO [04-03|21:04:35.636] Forkchoice requested sync to new head    number=16,970,410 hash=32d99f..cf698a finalized=unknown
WARN [04-03|21:04:35.636] Beacon chain gapped                      head=16,970,407 newHead=16,970,410
WARN [04-03|21:04:35.684] Ignoring payload while snap syncing      number=16,970,414 hash=2e570b..0b2712
WARN [04-03|21:04:35.770] Ignoring payload while snap syncing      number=16,970,415 hash=7ff9f8..573539
WARN [04-03|21:04:35.846] Ignoring payload while snap syncing      number=16,970,416 hash=121857..6455c9
WARN [04-03|21:04:35.939] Ignoring payload while snap syncing      number=16,970,417 hash=f1dba7..f31a03
WARN [04-03|21:04:36.005] Ignoring payload while snap syncing      number=16,970,418 hash=f63216..8f0384
WARN [04-03|21:04:36.138] Ignoring payload while snap syncing      number=16,970,419 hash=f83954..08e62f
WARN [04-03|21:04:36.210] Ignoring payload while snap syncing      number=16,970,420 hash=199f31..d6fa80
WARN [04-03|21:04:36.325] Ignoring payload while snap syncing      number=16,970,421 hash=9e7634..5ef1e2
WARN [04-03|21:04:36.438] Ignoring payload while snap syncing      number=16,970,422 hash=14fbdf..1c056b
WARN [04-03|21:04:36.526] Ignoring payload while snap syncing      number=16,970,423 hash=55432f..e5eff5
WARN [04-03|21:04:36.603] Ignoring payload while snap syncing      number=16,970,424 hash=b23f17..221182
WARN [04-03|21:04:36.660] Ignoring payload while snap syncing      number=16,970,425 hash=61b154..7985d9
WARN [04-03|21:04:36.747] Ignoring payload while snap syncing      number=16,970,426 hash=5e5eb6..7bfa3d
WARN [04-03|21:04:36.857] Ignoring payload while snap syncing      number=16,970,427 hash=7ce7f9..d59430
WARN [04-03|21:04:36.942] Ignoring payload while snap syncing      number=16,970,428 hash=21dc0a..892738
WARN [04-03|21:04:37.038] Ignoring payload while snap syncing      number=16,970,429 hash=6d94db..d1196a
WARN [04-03|21:04:37.111] Ignoring payload while snap syncing      number=16,970,430 hash=83ea5a..a65e0a
WARN [04-03|21:04:37.165] Ignoring payload while snap syncing      number=16,970,431 hash=ca464f..689da3
WARN [04-03|21:04:37.213] Ignoring payload while snap syncing      number=16,970,432 hash=0a1d30..8fa585
WARN [04-03|21:04:37.335] Ignoring payload while snap syncing      number=16,970,433 hash=de1878..8d98f5
WARN [04-03|21:04:37.382] Ignoring payload while snap syncing      number=16,970,434 hash=8de240..5e708d
WARN [04-03|21:04:37.439] Ignoring payload while snap syncing      number=16,970,435 hash=9f533b..a27529
WARN [04-03|21:04:37.484] Ignoring payload while snap syncing      number=16,970,436 hash=611f39..bb61f0
WARN [04-03|21:04:37.555] Ignoring payload while snap syncing      number=16,970,437 hash=f6c207..ab8d5e
WARN [04-03|21:04:37.621] Ignoring payload while snap syncing      number=16,970,438 hash=f4a35e..3cb9cd
WARN [04-03|21:04:37.831] Ignoring payload while snap syncing      number=16,970,439 hash=2e772d..b8bc9a
INFO [04-03|21:04:37.919] Looking for peers                        peercount=0 tried=33 static=0
INFO [04-03|21:04:38.465] Forkchoice requested sync to new head    number=16,970,439 hash=2e772d..b8bc9a finalized=unknown
WARN [04-03|21:04:38.465] Beacon chain gapped                      head=16,970,410 newHead=16,970,439
INFO [04-03|21:04:39.521] Forkchoice requested sync to new head    number=16,970,447 hash=fdafd2..118ecb finalized=unknown
INFO [04-03|21:04:39.875] Forkchoice requested sync to new head    number=16,970,448 hash=540ec4..a2e23a finalized=unknown
INFO [04-03|21:04:48.251] Looking for peers                        peercount=0 tried=35 static=0
INFO [04-03|21:04:50.632] Forkchoice requested sync to new head    number=16,970,449 hash=78fd0e..5aedf5 finalized=unknown
INFO [04-03|21:04:58.273] Looking for peers                        peercount=0 tried=33 static=0
INFO [04-03|21:05:00.398] Forkchoice requested sync to new head    number=16,970,450 hash=d42d4c..d4ba38 finalized=unknown
INFO [04-03|21:05:08.294] Looking for peers                        peercount=0 tried=32 static=0
INFO [04-03|21:05:14.525] Forkchoice requested sync to new head    number=16,970,451 hash=c78ef9..e5b92d finalized=unknown
INFO [04-03|21:05:18.874] Looking for peers                        peercount=0 tried=32 static=0
INFO [04-03|21:05:24.866] Forkchoice requested sync to new head    number=16,970,452 hash=adbaf1..f3e9f0 finalized=unknown
INFO [04-03|21:05:28.904] Looking for peers                        peercount=0 tried=43 static=0
INFO [04-03|21:05:35.828] Forkchoice requested sync to new head    number=16,970,453 hash=2f9b40..51d74c finalized=unknown
INFO [04-03|21:05:38.934] Looking for peers                        peercount=0 tried=38 static=0
INFO [04-03|21:05:47.931] Forkchoice requested sync to new head    number=16,970,454 hash=26bd25..5fc6d2 finalized=unknown
INFO [04-03|21:05:49.887] Looking for peers                        peercount=0 tried=34 static=0
INFO [04-03|21:05:59.901] Looking for peers                        peercount=0 tried=36 static=0
INFO [04-03|21:06:01.302] Forkchoice requested sync to new head    number=16,970,455 hash=952a98..d05f90 finalized=unknown
INFO [04-03|21:06:10.075] Looking for peers                        peercount=0 tried=41 static=0
INFO [04-03|21:06:12.980] Forkchoice requested sync to new head    number=16,970,456 hash=348f85..1ad838 finalized=unknown
INFO [04-03|21:06:20.237] Looking for peers                        peercount=0 tried=38 static=0
INFO [04-03|21:06:24.624] Forkchoice requested sync to new head    number=16,970,457 hash=ce0bca..65c612 finalized=unknown
INFO [04-03|21:06:30.900] Looking for peers                        peercount=0 tried=36 static=0
INFO [04-03|21:06:35.825] Forkchoice requested sync to new head    number=16,970,458 hash=b34bfb..d67f60 finalized=unknown
INFO [04-03|21:06:41.117] Looking for peers                        peercount=0 tried=46 static=0
INFO [04-03|21:06:48.707] Forkchoice requested sync to new head    number=16,970,459 hash=bffdad..e32d02 finalized=unknown
INFO [04-03|21:06:51.176] Looking for peers                        peercount=0 tried=43 static=0
INFO [04-03|21:07:01.218] Looking for peers                        peercount=0 tried=41 static=0
INFO [04-03|21:07:01.397] Forkchoice requested sync to new head    number=16,970,460 hash=a867b0..e75d76 finalized=unknown
INFO [04-03|21:07:11.226] Looking for peers                        peercount=0 tried=36 static=0
INFO [04-03|21:07:12.084] Forkchoice requested sync to new head    number=16,970,461 hash=180f70..3d68d2 finalized=unknown
INFO [04-03|21:07:21.579] Looking for peers                        peercount=0 tried=32 static=0
INFO [04-03|21:07:24.885] Forkchoice requested sync to new head    number=16,970,462 hash=d991d6..da8f33 finalized=unknown
INFO [04-03|21:07:32.238] Looking for peers                        peercount=0 tried=35 static=0
INFO [04-03|21:07:36.606] Forkchoice requested sync to new head    number=16,970,463 hash=ec6b16..81b4d4 finalized=unknown
INFO [04-03|21:07:42.270] Looking for peers                        peercount=0 tried=43 static=0
INFO [04-03|21:07:48.328] Forkchoice requested sync to new head    number=16,970,464 hash=5065c8..d246c8 finalized=unknown
INFO [04-03|21:07:52.544] Looking for peers                        peercount=0 tried=41 static=0
INFO [04-03|21:08:00.542] Forkchoice requested sync to new head    number=16,970,465 hash=9c21ef..56fe48 finalized=unknown
INFO [04-03|21:08:02.580] Looking for peers                        peercount=0 tried=33 static=0
INFO [04-03|21:08:12.586] Looking for peers                        peercount=0 tried=34 static=0
INFO [04-03|21:08:12.811] Forkchoice requested sync to new head    number=16,970,466 hash=52d163..ad69ba finalized=unknown
INFO [04-03|21:08:22.905] Looking for peers                        peercount=0 tried=25 static=0
INFO [04-03|21:08:24.090] Forkchoice requested sync to new head    number=16,970,467 hash=11ffff..10d6e0 finalized=unknown
INFO [04-03|21:08:25.238] Syncing beacon headers                   downloaded=512 left=16,969,926 eta=2117h2m40.078s
INFO [04-03|21:08:32.956] Looking for peers                        peercount=2 tried=45 static=0
INFO [04-03|21:08:33.860] Syncing beacon headers                   downloaded=26624 left=16,943,814 eta=42h10m26.286s
INFO [04-03|21:08:37.668] Forkchoice requested sync to new head    number=16,970,468 hash=f7a314..391d80 finalized=unknown
WARN [04-03|21:08:38.858] Served eth_coinbase                      reqid=3 duration="38.917µs" err="etherbase must be explicitly specified"
INFO [04-03|21:08:42.060] Syncing beacon headers                   downloaded=63488 left=16,906,950 eta=18h15m14.210s
INFO [04-03|21:08:43.171] Looking for peers                        peercount=2 tried=37 static=0
INFO [04-03|21:08:49.139] Forkchoice requested sync to new head    number=16,970,469 hash=8fd860..693e2e finalized=unknown
INFO [04-03|21:08:50.074] Syncing beacon headers                   downloaded=121,344 left=16,849,094 eta=9h49m37.184s
INFO [04-03|21:08:53.207] Looking for peers                        peercount=2 tried=39 static=0
INFO [04-03|21:08:58.156] Syncing beacon headers                   downloaded=207,360 left=16,763,078 eta=5h54m9.889s
INFO [04-03|21:09:01.028] Forkchoice requested sync to new head    number=16,970,470 hash=2a55d9..6e8e15 finalized=unknown
INFO [04-03|21:09:03.224] Looking for peers                        peercount=2 tried=40 static=0
INFO [04-03|21:09:06.225] Syncing beacon headers                   downloaded=288,256 left=16,682,182 eta=4h21m19.552s
INFO [04-03|21:09:10.531] Forkchoice requested sync to new head    number=16,970,470 hash=2a55d9..6e8e15 finalized=16,970,407
INFO [04-03|21:09:13.491] Forkchoice requested sync to new head    number=16,970,471 hash=b524a9..35f880 finalized=16,970,407
INFO [04-03|21:09:13.837] Looking for peers                        peercount=2 tried=33 static=0
INFO [04-03|21:09:14.249] Syncing beacon headers                   downloaded=370,688 left=16,599,750 eta=3h28m11.903s
INFO [04-03|21:09:22.290] Syncing beacon headers                   downloaded=446,464 left=16,523,974 eta=2h57m1.954s
INFO [04-03|21:09:23.732] Forkchoice requested sync to new head    number=16,970,472 hash=020a67..b46a76 finalized=16,970,407
INFO [04-03|21:09:24.166] Looking for peers                        peercount=2 tried=38 static=0
INFO [04-03|21:09:30.344] Syncing beacon headers                   downloaded=508,928 left=16,461,510 eta=2h39m3.543s
INFO [04-03|21:09:34.172] Looking for peers                        peercount=2 tried=44 static=0
INFO [04-03|21:09:37.342] Forkchoice requested sync to new head    number=16,970,473 hash=a1bec1..4fb0bf finalized=16,970,407
INFO [04-03|21:09:38.412] Syncing beacon headers                   downloaded=572,416 left=16,398,022 eta=2h24m43.457s
INFO [04-03|21:09:46.517] Syncing beacon headers                   downloaded=640,000 left=16,330,438 eta=2h12m21.278s
INFO [04-03|21:09:48.467] Forkchoice requested sync to new head    number=16,970,474 hash=74dcd9..7a8d25 finalized=16,970,407
INFO [04-03|21:09:54.503] Looking for peers                        peercount=2 tried=40 static=0
INFO [04-03|21:09:54.556] Syncing beacon headers                   downloaded=715,776 left=16,254,662 eta=2h0m50.192s
INFO [04-03|21:10:01.485] Forkchoice requested sync to new head    number=16,970,475 hash=dd68ab..c242aa finalized=16,970,407
INFO [04-03|21:10:02.691] Syncing beacon headers                   downloaded=800,256 left=16,170,182 eta=1h50m15.501s
INFO [04-03|21:10:04.504] Looking for peers                        peercount=2 tried=36 static=0
INFO [04-03|21:10:10.781] Syncing beacon headers                   downloaded=880,128 left=16,090,310 eta=1h42m13.333s
INFO [04-03|21:10:12.477] Forkchoice requested sync to new head    number=16,970,476 hash=8c80ab..52666e finalized=16,970,407
INFO [04-03|21:10:14.655] Looking for peers                        peercount=2 tried=33 static=0
INFO [04-03|21:10:18.876] Syncing beacon headers                   downloaded=967,680 left=16,002,758 eta=1h34m41.918s
ERROR[04-03|21:10:20.407] Cleaning spurious beacon sync leftovers  stale_head_1=15,989,135 stale_tail_1=15,838,462 stale_next_1=ab0b4e..b9041f
INFO [04-03|21:10:20.408] Syncing beacon headers                   downloaded=981,404 left=0          eta=0s
INFO [04-03|21:10:20.408] Block synchronisation started
INFO [04-03|21:10:20.517] Syncing: state healing in progress       accounts=0@0.00B slots=0@0.00B codes=401@3.24MiB nodes=6,222,565@1.42GiB pending=17
INFO [04-03|21:10:20.793] Syncing: chain download in progress      synced=94.22% chain=392.63GiB headers=16,007,466@7.19GiB bodies=15,989,035@261.52GiB receipts=15,989,035@123.92GiB eta=78h0m48.624s
INFO [04-03|21:10:24.243] Forkchoice requested sync to new head    number=16,970,477 hash=ea3dc5..a6bef0 finalized=16,970,407
INFO [04-03|21:10:24.709] Looking for peers                        peercount=2 tried=35 static=0
INFO [04-03|21:10:28.524] Syncing: state healing in progress       accounts=0@0.00B slots=0@0.00B codes=401@3.24MiB nodes=6,222,653@1.42GiB pending=1057
INFO [04-03|21:10:29.085] Syncing: chain download in progress      synced=94.22% chain=392.66GiB headers=16,025,898@7.19GiB bodies=15,989,579@261.55GiB receipts=15,989,579@123.92GiB eta=4h17m20.053s
INFO [04-03|21:10:36.435] Forkchoice requested sync to new head    number=16,970,478 hash=2d806a..30ee29 finalized=16,970,407
INFO [04-03|21:10:36.533] Syncing: state healing in progress       accounts=3@131.00B slots=0@0.00B codes=401@3.24MiB nodes=6,223,225@1.42GiB pending=5177
INFO [04-03|21:10:37.207] Syncing: chain download in progress      synced=94.22% chain=392.70GiB headers=16,025,898@7.19GiB bodies=15,990,265@261.58GiB receipts=15,990,265@123.94GiB eta=3h41m38.035s
INFO [04-03|21:10:44.667] Syncing: state healing in progress       accounts=138@6.48KiB slots=0@0.00B codes=401@3.24MiB nodes=6,225,173@1.42GiB pending=11287
INFO [04-03|21:10:44.748] Looking for peers                        peercount=2 tried=39 static=0
INFO [04-03|21:10:45.864] Syncing: chain download in progress      synced=94.23% chain=392.74GiB headers=16,027,946@7.19GiB bodies=15,990,748@261.61GiB receipts=15,990,748@123.94GiB eta=4h1m34.730s
INFO [04-03|21:10:48.039] Forkchoice requested sync to new head    number=16,970,479 hash=299145..b1ce8b finalized=16,970,407
INFO [04-03|21:10:53.195] Syncing: state healing in progress       accounts=1313@63.86KiB slots=9@739.00B codes=401@3.24MiB nodes=6,230,122@1.42GiB pending=13243
INFO [04-03|21:10:54.835] Looking for peers                        peercount=2 tried=36 static=0
INFO [04-03|21:10:54.947] Syncing: chain download in progress      synced=94.23% chain=392.78GiB headers=16,027,946@7.19GiB bodies=15,991,457@261.64GiB receipts=15,991,457@123.96GiB eta=3h51m56.118s
INFO [04-03|21:11:00.537] Forkchoice requested sync to new head    number=16,970,480 hash=c15f24..a4c8a9 finalized=16,970,407
INFO [04-03|21:11:01.866] Syncing: state healing in progress       accounts=2716@132.63KiB slots=9@739.00B codes=401@3.24MiB nodes=6,235,885@1.43GiB pending=14771
INFO [04-03|21:11:03.107] Syncing: chain download in progress      synced=94.24% chain=392.83GiB headers=16,027,946@7.19GiB bodies=15,992,237@261.67GiB receipts=15,992,237@123.97GiB eta=3h36m50.864s
INFO [04-03|21:11:04.885] Looking for peers                        peercount=2 tried=37 static=0
INFO [04-03|21:11:10.155] Syncing: state healing in progress       accounts=4755@231.96KiB slots=9@739.00B codes=401@3.24MiB nodes=6,242,684@1.43GiB pending=14896
INFO [04-03|21:11:11.121] Syncing: chain download in progress      synced=94.24% chain=392.87GiB headers=16,029,994@7.19GiB bodies=15,993,092@261.70GiB receipts=15,993,092@123.98GiB eta=3h23m10.976s
INFO [04-03|21:11:13.095] Forkchoice requested sync to new head    number=16,970,481 hash=e8d490..8b1e60 finalized=16,970,407
INFO [04-03|21:11:18.525] Syncing: state healing in progress       accounts=6281@306.40KiB slots=9@739.00B codes=401@3.24MiB nodes=6,248,266@1.43GiB pending=15327
INFO [04-03|21:11:19.186] Syncing: chain download in progress      synced=94.24% chain=392.91GiB headers=16,029,994@7.19GiB bodies=15,993,738@261.73GiB receipts=15,993,738@123.99GiB eta=3h23m4.357s
INFO [04-03|21:11:24.434] Forkchoice requested sync to new head    number=16,970,482 hash=a49d66..4e3ef4 finalized=16,970,407
INFO [04-03|21:11:25.167] Looking for peers                        peercount=2 tried=45 static=0
INFO [04-03|21:11:26.839] Syncing: state healing in progress       accounts=8033@390.98KiB slots=9@739.00B codes=401@3.24MiB nodes=6,254,426@1.43GiB pending=15306
INFO [04-03|21:11:27.248] Syncing: chain download in progress      synced=94.25% chain=392.95GiB headers=16,029,994@7.19GiB bodies=15,994,546@261.75GiB receipts=15,994,546@124.00GiB eta=3h16m57.245s
INFO [04-03|21:11:35.041] Syncing: state healing in progress       accounts=9727@473.93KiB slots=9@739.00B codes=402@3.24MiB nodes=6,260,239@1.43GiB pending=15153
INFO [04-03|21:11:35.252] Syncing: chain download in progress      synced=94.25% chain=392.99GiB headers=16,032,042@7.19GiB bodies=15,995,356@261.78GiB receipts=15,995,356@124.01GiB eta=3h12m9.163s
INFO [04-03|21:11:35.497] Looking for peers                        peercount=2 tried=38 static=0
INFO [04-03|21:11:36.073] Forkchoice requested sync to new head    number=16,970,483 hash=94533e..080ec1 finalized=16,970,407
INFO [04-03|21:11:43.057] Syncing: state healing in progress       accounts=11625@566.59KiB slots=9@739.00B codes=402@3.24MiB nodes=6,266,729@1.43GiB pending=15347
INFO [04-03|21:11:43.450] Syncing: chain download in progress      synced=94.26% chain=393.04GiB headers=16,032,042@7.19GiB bodies=15,996,231@261.82GiB receipts=15,996,231@124.03GiB eta=3h7m8.182s
INFO [04-03|21:11:45.500] Looking for peers                        peercount=2 tried=42 static=0
INFO [04-03|21:11:48.381] Forkchoice requested sync to new head    number=16,970,484 hash=68c783..e8831d finalized=16,970,407
INFO [04-03|21:11:51.125] Syncing: state healing in progress       accounts=13152@641.85KiB slots=9@739.00B codes=402@3.24MiB nodes=6,271,832@1.44GiB pending=15004
INFO [04-03|21:11:51.545] Syncing: chain download in progress      synced=94.26% chain=393.07GiB headers=16,032,042@7.19GiB bodies=15,996,776@261.84GiB receipts=15,996,776@124.03GiB eta=3h10m50.046s
INFO [04-03|21:11:59.275] Syncing: state healing in progress       accounts=14936@727.98KiB slots=9@739.00B codes=402@3.24MiB nodes=6,278,109@1.44GiB pending=15350
INFO [04-03|21:12:00.925] Forkchoice requested sync to new head    number=16,970,485 hash=7a0c84..ed2f56 finalized=16,970,407
INFO [04-03|21:12:01.120] Syncing: chain download in progress      synced=94.27% chain=393.11GiB headers=16,034,090@7.19GiB bodies=15,997,516@261.88GiB receipts=15,997,516@124.05GiB eta=3h12m21.503s
INFO [04-03|21:12:05.832] Looking for peers                        peercount=2 tried=33 static=0
INFO [04-03|21:12:07.354] Syncing: state healing in progress       accounts=16618@810.45KiB slots=9@739.00B codes=402@3.24MiB nodes=6,283,874@1.44GiB pending=15529
INFO [04-03|21:12:09.206] Syncing: chain download in progress      synced=94.27% chain=393.17GiB headers=16,034,090@7.19GiB bodies=15,998,408@261.92GiB receipts=15,998,408@124.06GiB eta=3h7m52.210s
INFO [04-03|21:12:12.510] Forkchoice requested sync to new head    number=16,970,486 hash=9fb5f5..2769ff finalized=16,970,407
INFO [04-03|21:12:15.543] Syncing: state healing in progress       accounts=18218@888.85KiB slots=9@739.00B codes=402@3.24MiB nodes=6,289,402@1.44GiB pending=15423
INFO [04-03|21:12:15.833] Looking for peers                        peercount=2 tried=38 static=0
INFO [04-03|21:12:17.648] Syncing: chain download in progress      synced=94.27% chain=393.20GiB headers=16,036,138@7.19GiB bodies=15,998,925@261.94GiB receipts=15,998,925@124.07GiB eta=3h11m46.625s
INFO [04-03|21:12:24.068] Syncing: state healing in progress       accounts=19647@957.82KiB slots=9@739.00B codes=402@3.24MiB nodes=6,294,182@1.44GiB pending=15309
INFO [04-03|21:12:25.024] Forkchoice requested sync to new head    number=16,970,487 hash=4d24b6..797628 finalized=16,970,407
INFO [04-03|21:12:25.886] Looking for peers                        peercount=2 tried=39 static=0
INFO [04-03|21:12:27.941] Syncing: chain download in progress      synced=94.28% chain=393.23GiB headers=16,036,138@7.19GiB bodies=15,999,611@261.96GiB receipts=15,999,611@124.08GiB eta=3h14m57.546s
INFO [04-03|21:12:32.396] Syncing: state healing in progress       accounts=21446@1.02MiB   slots=9@739.00B codes=403@3.26MiB nodes=6,300,317@1.44GiB pending=15587
INFO [04-03|21:12:36.023] Syncing: chain download in progress      synced=94.28% chain=393.27GiB headers=16,036,138@7.19GiB bodies=16,000,382@261.99GiB receipts=16,000,382@124.09GiB eta=3h13m5.040s
INFO [04-03|21:12:36.081] Looking for peers                        peercount=2 tried=41 static=0
INFO [04-03|21:12:36.455] Forkchoice requested sync to new head    number=16,970,488 hash=b32f47..ec45ad finalized=16,970,407
INFO [04-03|21:12:40.444] Syncing: state healing in progress       accounts=23004@1.10MiB   slots=127@10.11KiB codes=403@3.26MiB nodes=6,306,134@1.44GiB pending=16801
INFO [04-03|21:12:44.253] Syncing: chain download in progress      synced=94.29% chain=393.32GiB headers=16,038,186@7.20GiB bodies=16,001,473@262.02GiB receipts=16,001,473@124.10GiB eta=3h6m38.208s
INFO [04-03|21:12:48.278] Forkchoice requested sync to new head    number=16,970,489 hash=6b6caa..38ce25 finalized=16,970,407
INFO [04-03|21:12:48.852] Syncing: state healing in progress       accounts=24650@1.17MiB   slots=333@27.02KiB codes=403@3.26MiB nodes=6,313,097@1.45GiB pending=17462
INFO [04-03|21:12:52.996] Syncing: chain download in progress      synced=94.30% chain=393.41GiB headers=16,040,234@7.20GiB bodies=16,003,250@262.09GiB receipts=16,003,250@124.13GiB eta=2h52m55.354s
INFO [04-03|21:12:56.958] Syncing: state healing in progress       accounts=26346@1.25MiB   slots=507@40.78KiB codes=403@3.26MiB nodes=6,319,604@1.45GiB pending=17475
INFO [04-03|21:13:00.948] Forkchoice requested sync to new head    number=16,970,490 hash=e36cc3..3f8e30 finalized=16,970,407
INFO [04-03|21:13:02.046] Syncing: chain download in progress      synced=94.31% chain=393.52GiB headers=16,042,282@7.20GiB bodies=16,004,882@262.17GiB receipts=16,004,882@124.15GiB eta=2h44m2.655s
INFO [04-03|21:13:05.331] Syncing: state healing in progress       accounts=27950@1.33MiB   slots=753@59.46KiB codes=403@3.26MiB nodes=6,325,709@1.45GiB pending=17111
INFO [04-03|21:13:10.142] Syncing: chain download in progress      synced=94.32% chain=393.62GiB headers=16,042,282@7.20GiB bodies=16,006,444@262.24GiB receipts=16,006,444@124.18GiB eta=2h36m33.401s
INFO [04-03|21:13:12.896] Forkchoice requested sync to new head    number=16,970,491 hash=9f8956..3e3be7 finalized=16,970,407
INFO [04-03|21:13:13.557] Syncing: state healing in progress       accounts=29701@1.41MiB   slots=874@68.57KiB codes=403@3.26MiB nodes=6,332,278@1.45GiB pending=18146
INFO [04-03|21:13:18.239] Syncing: chain download in progress      synced=94.33% chain=393.70GiB headers=16,044,330@7.20GiB bodies=16,007,993@262.30GiB receipts=16,007,993@124.20GiB eta=2h30m23.162s
INFO [04-03|21:13:21.970] Syncing: state healing in progress       accounts=31300@1.49MiB   slots=997@77.80KiB codes=403@3.26MiB nodes=6,338,578@1.45GiB pending=19504
INFO [04-03|21:13:24.026] Forkchoice requested sync to new head    number=16,970,492 hash=d87a6c..edd225 finalized=16,970,407
INFO [04-03|21:13:27.105] Syncing: chain download in progress      synced=94.34% chain=393.79GiB headers=16,046,378@7.20GiB bodies=16,009,657@262.37GiB receipts=16,009,657@124.22GiB eta=2h24m53.823s
INFO [04-03|21:13:30.127] Syncing: state healing in progress       accounts=33214@1.58MiB   slots=1091@84.71KiB codes=403@3.26MiB nodes=6,345,262@1.45GiB pending=19629
INFO [04-03|21:13:36.413] Syncing: chain download in progress      synced=94.35% chain=393.88GiB headers=16,048,426@7.20GiB bodies=16,011,198@262.44GiB receipts=16,011,198@124.24GiB eta=2h21m19.269s
INFO [04-03|21:13:37.020] Forkchoice requested sync to new head    number=16,970,493 hash=e7006c..01f838 finalized=16,970,407
INFO [04-03|21:13:38.127] Syncing: state healing in progress       accounts=34755@1.66MiB   slots=1898@149.20KiB codes=403@3.26MiB nodes=6,352,592@1.46GiB pending=20423
INFO [04-03|21:13:44.629] Syncing: chain download in progress      synced=94.36% chain=394.00GiB headers=16,048,426@7.20GiB bodies=16,012,956@262.53GiB receipts=16,012,956@124.27GiB eta=2h16m10.645s
INFO [04-03|21:13:46.512] Syncing: state healing in progress       accounts=34818@1.66MiB   slots=6130@496.62KiB codes=403@3.26MiB nodes=6,362,740@1.46GiB pending=22210
INFO [04-03|21:13:48.176] Forkchoice requested sync to new head    number=16,970,494 hash=7f1b06..c320c1 finalized=16,970,407
INFO [04-03|21:13:53.133] Syncing: chain download in progress      synced=94.36% chain=394.07GiB headers=16,050,474@7.20GiB bodies=16,014,138@262.58GiB receipts=16,014,138@124.29GiB eta=2h15m0.283s
INFO [04-03|21:13:54.694] Syncing: state healing in progress       accounts=34818@1.66MiB   slots=9264@762.25KiB codes=403@3.26MiB nodes=6,372,074@1.46GiB pending=20534
INFO [04-03|21:14:00.731] Forkchoice requested sync to new head    number=16,970,495 hash=970579..4e86d5 finalized=16,970,407
INFO [04-03|21:14:01.341] Syncing: chain download in progress      synced=94.38% chain=394.20GiB headers=16,052,522@7.20GiB bodies=16,016,417@262.67GiB receipts=16,016,417@124.33GiB eta=2h8m14.426s
INFO [04-03|21:14:02.818] Syncing: state healing in progress       accounts=35322@1.68MiB   slots=11495@943.96KiB codes=403@3.26MiB nodes=6,380,324@1.46GiB pending=18467
INFO [04-03|21:14:11.013] Syncing: state healing in progress       accounts=36366@1.73MiB   slots=13649@1.09MiB   codes=403@3.26MiB nodes=6,389,271@1.46GiB pending=17315
INFO [04-03|21:14:11.178] Syncing: chain download in progress      synced=94.39% chain=394.34GiB headers=16,054,570@7.20GiB bodies=16,018,688@262.78GiB receipts=16,018,688@124.36GiB eta=2h3m23.993s
INFO [04-03|21:14:12.571] Forkchoice requested sync to new head    number=16,970,496 hash=cb3a12..940c29 finalized=16,970,407
INFO [04-03|21:14:19.411] Syncing: state healing in progress       accounts=36863@1.76MiB   slots=17033@1.33MiB   codes=403@3.26MiB nodes=6,398,947@1.47GiB pending=16733
INFO [04-03|21:14:20.155] Syncing: chain download in progress      synced=94.40% chain=394.47GiB headers=16,056,618@7.20GiB bodies=16,020,539@262.87GiB receipts=16,020,539@124.39GiB eta=2h0m26.145s
INFO [04-03|21:14:24.180] Forkchoice requested sync to new head    number=16,970,497 hash=304103..4ec511 finalized=16,970,407
INFO [04-03|21:14:27.764] Syncing: state healing in progress       accounts=38431@1.83MiB   slots=17894@1.39MiB   codes=403@3.26MiB nodes=6,406,537@1.47GiB pending=16688
INFO [04-03|21:14:28.828] Syncing: chain download in progress      synced=94.41% chain=394.57GiB headers=16,058,666@7.21GiB bodies=16,022,332@262.94GiB receipts=16,022,332@124.42GiB eta=1h57m51.096s
INFO [04-03|21:14:36.158] Syncing: state healing in progress       accounts=39974@1.91MiB   slots=18681@1.45MiB   codes=403@3.26MiB nodes=6,414,203@1.47GiB pending=19032
INFO [04-03|21:14:36.974] Syncing: chain download in progress      synced=94.42% chain=394.64GiB headers=16,060,714@7.21GiB bodies=16,023,700@263.00GiB receipts=16,023,700@124.44GiB eta=1h56m44.739s
INFO [04-03|21:14:37.123] Forkchoice requested sync to new head    number=16,970,498 hash=4129e0..1ce456 finalized=16,970,407
INFO [04-03|21:14:44.399] Syncing: state healing in progress       accounts=41266@1.97MiB   slots=19646@1.53MiB   codes=403@3.26MiB nodes=6,421,876@1.47GiB pending=18866
INFO [04-03|21:14:45.129] Syncing: chain download in progress      synced=94.43% chain=394.77GiB headers=16,062,762@7.21GiB bodies=16,025,646@263.09GiB receipts=16,025,646@124.47GiB eta=1h53m49.273s
INFO [04-03|21:14:48.889] Forkchoice requested sync to new head    number=16,970,499 hash=5d150e..730b4d finalized=16,970,407
INFO [04-03|21:14:52.561] Syncing: state healing in progress       accounts=42321@2.02MiB   slots=21198@1.64MiB   codes=403@3.26MiB nodes=6,430,692@1.47GiB pending=23808
INFO [04-03|21:14:53.298] Syncing: chain download in progress      synced=94.44% chain=394.87GiB headers=16,062,762@7.21GiB bodies=16,027,139@263.17GiB receipts=16,027,139@124.50GiB eta=1h52m33.556s
INFO [04-03|21:15:00.983] Syncing: state healing in progress       accounts=42798@2.04MiB   slots=27897@2.11MiB   codes=403@3.26MiB nodes=6,444,703@1.48GiB pending=17784
INFO [04-03|21:15:01.031] Forkchoice requested sync to new head    number=16,970,500 hash=3d84e0..8924a8 finalized=16,970,407
INFO [04-03|21:15:01.589] Syncing: chain download in progress      synced=94.45% chain=394.98GiB headers=16,066,858@7.21GiB bodies=16,029,035@263.24GiB receipts=16,029,035@124.53GiB eta=1h50m15.676s
INFO [04-03|21:15:08.987] Syncing: state healing in progress       accounts=44353@2.11MiB   slots=28366@2.14MiB   codes=403@3.26MiB nodes=6,452,104@1.48GiB pending=18498
INFO [04-03|21:15:09.744] Syncing: chain download in progress      synced=94.46% chain=395.07GiB headers=16,066,858@7.21GiB bodies=16,030,601@263.31GiB receipts=16,030,601@124.55GiB eta=1h49m0.246s
INFO [04-03|21:15:12.294] Forkchoice requested sync to new head    number=16,970,501 hash=3ef77f..2058fb finalized=16,970,407
INFO [04-03|21:15:17.233] Syncing: state healing in progress       accounts=46352@2.21MiB   slots=28608@2.16MiB   codes=403@3.26MiB nodes=6,459,587@1.48GiB pending=18320
INFO [04-03|21:15:18.619] Syncing: chain download in progress      synced=94.48% chain=395.22GiB headers=16,068,906@7.21GiB bodies=16,033,128@263.42GiB receipts=16,033,128@124.59GiB eta=1h45m37.521s
INFO [04-03|21:15:24.854] Forkchoice requested sync to new head    number=16,970,502 hash=8f150b..554ce9 finalized=16,970,407
INFO [04-03|21:15:25.636] Syncing: state healing in progress       accounts=48061@2.29MiB   slots=28854@2.18MiB   codes=403@3.26MiB nodes=6,466,451@1.48GiB pending=19276
INFO [04-03|21:15:26.872] Syncing: chain download in progress      synced=94.49% chain=395.34GiB headers=16,070,954@7.21GiB bodies=16,034,687@263.51GiB receipts=16,034,687@124.62GiB eta=1h44m40.118s
INFO [04-03|21:15:33.707] Syncing: state healing in progress       accounts=48356@2.31MiB   slots=30903@2.33MiB   codes=403@3.26MiB nodes=6,474,154@1.48GiB pending=19123
INFO [04-03|21:15:34.523] Forkchoice requested sync to new head    number=16,970,502 hash=8f150b..554ce9 finalized=16,970,439
INFO [04-03|21:15:35.010] Syncing: chain download in progress      synced=94.49% chain=395.42GiB headers=16,073,002@7.21GiB bodies=16,036,097@263.57GiB receipts=16,036,097@124.64GiB eta=1h44m4.349s
INFO [04-03|21:15:39.978] Forkchoice requested sync to new head    number=16,970,503 hash=e6a001..92bdbc finalized=16,970,439
INFO [04-03|21:15:42.030] Syncing: state healing in progress       accounts=48611@2.32MiB   slots=32900@2.48MiB   codes=403@3.26MiB nodes=6,482,625@1.48GiB pending=26104
INFO [04-03|21:15:43.215] Syncing: chain download in progress      synced=94.51% chain=395.53GiB headers=16,075,050@7.21GiB bodies=16,038,174@263.65GiB receipts=16,038,174@124.67GiB eta=1h42m2.809s
cINFO [04-03|21:15:48.884] Forkchoice requested sync to new head    number=16,970,504 hash=7f8770..a3696a finalized=16,970,439
INFO [04-03|21:15:50.389] Syncing: state healing in progress       accounts=48611@2.32MiB   slots=39097@2.96MiB   codes=403@3.26MiB nodes=6,495,781@1.49GiB pending=23018
INFO [04-03|21:15:51.269] Syncing: chain download in progress      synced=94.52% chain=395.62GiB headers=16,077,098@7.21GiB bodies=16,039,741@263.72GiB receipts=16,039,741@124.69GiB eta=1h41m11.478s