Geth stuck on 100%: Syncing: chain download in progress, Validator down since Dencun :(

[aviadshimoni]

System information

Geth version: 1.13.14-stable

CL client & version: prysm v5
OS & Version: Ubuntu 22.04 LTS
I''ll start with a thanks to the community, maintaining an open source project in such scale is no joke.
Since Dencun update, my validator is off.
I'm using the guide from here:
https://github.com/metanull-operator/eth2-ubuntu/blob/master/v2/setup.md
It was working quite decent in the last couple of years, the Hardware specs (I do think relevant) are:

It was working quite good in the last couple of years, the Hardware specs (I do think relevant) are:

• https://www.amazon.com/Beelink-Windows-i5-8279U-Graphics-Computer/dp/B0967CJ3G7
• Upgraded nvme` to XPG SX8200 Pro 2TB
• Upgraded RAM to 32GB
  total used free shared buff/cache available
  Mem: 31Gi 14Gi 4.1Gi 570Mi 13Gi 16Gi
  Swap: 2.0Gi 1.0Mi 2.0Gi
  aviad@aviad-SEi:~$

Expected behaviour

I've tried to start from scratch (deleting disk, installing ubuntu, bootstrap, etc) several times, but no luck, It always come to this part when I suspect that my beacon or validator is down.

Actual behaviour

Geth stuck on the same block:

eth.syncing
{
currentBlock: 19552677,
healedBytecodeBytes: 529932,
healedBytecodes: 86,
healedTrienodeBytes: 550398257,
healedTrienodes: 1948263,
healingBytecode: 0,
healingTrienodes: 1,
highestBlock: 19552736,
startingBlock: 14268309,
syncedAccountBytes: 59739193788,
syncedAccounts: 244464929,
syncedBytecodeBytes: 8243652824,
syncedBytecodes: 1176020,
syncedStorage: 1111573870,
syncedStorageBytes: 239830071225,
txIndexFinishedBlocks: 2350000,
txIndexRemainingBlocks: 0
}

Geth logs:

Apr 01 00:27:33 aviad-SEi geth[46773]: INFO [04-01|00:27:33.665] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,552,736@9.08GiB bodies=19,552,677@535.19GiB receipts=19,552,677@184.32GiB eta=914.153ms
Apr 01 00:27:41 aviad-SEi geth[46773]: INFO [04-01|00:27:41.671] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,552,736@9.08GiB bodies=19,552,677@535.19GiB receipts=19,552,677@184.32GiB eta=914.242ms
Apr 01 00:27:49 aviad-SEi geth[46773]: INFO [04-01|00:27:49.675] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,552,736@9.08GiB bodies=19,552,677@535.19GiB receipts=19,552,677@184.32GiB eta=914.332ms
Apr 01 00:27:57 aviad-SEi geth[46773]: INFO [04-01|00:27:57.679] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,552,736@9.08GiB bodies=19,552,677@535.19GiB receipts=19,552,677@184.32GiB eta=914.421ms
Apr 01 00:28:05 aviad-SEi geth[46773]: INFO [04-01|00:28:05.685] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,552,736@9.08GiB bodies=19,552,677@535.19GiB receipts=19,552,677@184.32GiB eta=914.510ms
Apr 01 00:28:13 aviad-SEi geth[46773]: INFO [04-01|00:28:13.691] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,552,736@9.08GiB bodies=19,552,677@535.19GiB receipts=19,552,677@184.32GiB eta=914.600ms

Last thing, some logs from Prysm:

Apr 01 00:27:43 aviad-SEi prysm.sh[64776]: time="2024-04-01 00:27:43" level=warning msg="Execution client is not syncing" prefix=execution
Apr 01 00:27:46 aviad-SEi prysm.sh[64776]: time="2024-04-01 00:27:46" level=info msg="Peer summary" activePeers=16 inbound=0 outbound=0 prefix=p2p
Apr 01 00:27:57 aviad-SEi prysm.sh[64776]: time="2024-04-01 00:27:57" level=warning msg="Execution client is not syncing" prefix=execution
Apr 01 00:28:11 aviad-SEi prysm.sh[64776]: time="2024-04-01 00:28:11" level=warning msg="Execution client is not syncing" prefix=execution
Apr 01 00:28:25 aviad-SEi prysm.sh[64776]: time="2024-04-01 00:28:25" level=warning msg="Execution client is not syncing" prefix=execution
Apr 01 00:28:39 aviad-SEi prysm.sh[64776]: time="2024-04-01 00:28:39" level=warning msg="Execution client is not syncing" prefix=execution
Apr 01 00:28:46 aviad-SEi prysm.sh[64776]: time="2024-04-01 00:28:46" level=info msg="Peer summary" activePeers=9 inbound=0 outbound=0 prefix=p2p
Apr 01 00:28:53 aviad-SEi prysm.sh[64776]: time="2024-04-01 00:28:53" level=warning msg="Execution client is not syncing" prefix=execution
Apr 01 00:29:07 aviad-SEi prysm.sh[64776]: time="2024-04-01 00:29:07" level=warning msg="Execution client is not syncing" prefix=execution
Apr 01 00:29:21 aviad-SEi prysm.sh[64776]: time="2024-04-01 00:29:21" level=warning msg="Execution client is not syncing" prefix=execution
aviad@aviad-SEi:~$ 

Few questions to make it more specific because I did some debugging:

1. Hardware looks o.k?
2. How to enable more debugging with Geth, some useful commands except eth.syncing
3. I've tried to geth removedb, should I do it again? I didn't clear the db on prism though.
4. It's like this for ~3 days with snap sync, I've also increased the cache size to 8GB to enable less write to the disk.

Thanks in advance, I would happily engage in code writing if needed,

[rjl493456442]

Can you share a bit more logs? Looks like the state sync is not finished yet healingTrienodes: 1

[aviadshimoni]

More logs from prysm-beacon after enabling debug:
Apr 01 11:38:54 aviad-SEi prysm.sh[69571]: time="2024-04-01 11:38:54" level=debug msg="Aggregated unaggregated ```
attestations" duration=1.704888ms prefix="pool/attestations"
Apr 01 11:38:54 aviad-SEi prysm.sh[69571]: time="2024-04-01 11:38:54" level=info msg="Peer summary" activePeers=69 inbound=0 outbound=69 prefix=p2p
Apr 01 11:38:56 aviad-SEi prysm.sh[69571]: time="2024-04-01 11:38:56" level=debug msg="Sending blob by range request" count=62 prefix=sync startSlot=8761123 topic="/eth2/beacon_chain/req/blob_sidecars_by_range/1"
Apr 01 11:38:57 aviad-SEi prysm.sh[69571]: time="2024-04-01 11:38:57" level=debug msg="Requesting blobs" capacity=20 count=62 peer=16Uiu2HAmBJerTBUh9e1UrEGkNxaSoMqUYqn14BFRrrDuhehLY1e2 prefix=initial-sync score="[0.0%, raw: 0.00, blocks: 0/640]" start=8761187
Apr 01 11:38:57 aviad-SEi prysm.sh[69571]: time="2024-04-01 11:38:57" level=debug msg="Slowing down for rate limit" peer=16Uiu2HAmBJerTBUh9e1UrEGkNxaSoMqUYqn14BFRrrDuhehLY1e2 prefix=initial-sync
Apr 01 11:38:57 aviad-SEi prysm.sh[69571]: time="2024-04-01 11:38:57" level=debug msg="Latest eth1 chain event" blockHash=0x03b20abf81f9bccb5c24249f43875491461a7e168e798680a8a1b7c3b897cf81 blockNumber=19559501 prefix=execution
Apr 01 11:38:57 aviad-SEi prysm.sh[69571]: time="2024-04-01 11:38:57" level=warning msg="Execution client is not syncing" prefix=execution
Apr 01 11:38:57 aviad-SEi prysm.sh[69571]: time="2024-04-01 11:38:57" level=error msg="Got a validation error in newPayload" error="links to previously rejected block" prefix=execution
Apr 01 11:38:57 aviad-SEi prysm.sh[69571]: time="2024-04-01 11:38:57" level=warning msg="Pruned invalid blocks" blockRoot=0xc01691baa7ff6f10f4f87c1dd3a944c228568d1979544e0a92251a0b6f3e80f2 invalidChildrenCount=0 prefix=blockchain
Apr 01 11:38:57 aviad-SEi prysm.sh[69571]: time="2024-04-01 11:38:57" level=warning msg="Block processing failure" error="could not notify the engine of the new payload: received an INVALID payload from execution engine" firstSlot=8761122 firstUnprocessed=8761122 lastSlot=8761122 prefix=initial-sync root=0xc01691baa7ff6f10f4f87c1dd3a944c228568d1979544e0a92251a0b6f3e80f2

more logs from geth:

Apr 01 11:13:55 aviad-SEi geth[69608]: INFO [04-01|11:13:55.722] NAT mapped port proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 11:13:56 aviad-SEi geth[69608]: INFO [04-01|11:13:56.236] NAT mapped port proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 11:21:56 aviad-SEi geth[69608]: INFO [04-01|11:21:56.042] NAT mapped port proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 11:21:56 aviad-SEi geth[69608]: INFO [04-01|11:21:56.557] NAT mapped port proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 11:29:49 aviad-SEi geth[69608]: WARN [04-01|11:29:49.023] Beacon client online, but no consensus updates received in a while. Please fix your beacon client to follow the chain!
Apr 01 11:29:56 aviad-SEi geth[69608]: INFO [04-01|11:29:56.368] NAT mapped port proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 11:29:56 aviad-SEi geth[69608]: INFO [04-01|11:29:56.883] NAT mapped port proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 11:37:56 aviad-SEi geth[69608]: INFO [04-01|11:37:56.690] NAT mapped port proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 11:37:57 aviad-SEi geth[69608]: INFO [04-01|11:37:57.210] NAT mapped port proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 11:38:25 aviad-SEi geth[69608]: WARN [04-01|11:38:25.230] Served eth_coinbase conn=127.0.0.1:41904 reqid=3 duration="22.551µs" err="etherbase must be explicitly specified"

[aviadshimoni]

another eth.syncing output:

> eth.syncing
{
  currentBlock: 19560920,
  healedBytecodeBytes: 1695256,
  healedBytecodes: 251,
  healedTrienodeBytes: 1250409427,
  healedTrienodes: 4262683,
  healingBytecode: 0,
  healingTrienodes: 0,
  highestBlock: 19561000,
  startingBlock: 19551746,
  syncedAccountBytes: 59769308126,
  syncedAccounts: 243737631,
  syncedBytecodeBytes: 8436465705,
  syncedBytecodes: 1210234,
  syncedStorage: 1118792764,
  syncedStorageBytes: 241442409945,
  txIndexFinishedBlocks: 2350000,
  txIndexRemainingBlocks: 0
}

geth logs:

viad@aviad-SEi:~$ sudo journalctl -fu geth
[sudo] password for aviad: 
Apr 01 15:38:07 aviad-SEi geth[69608]: INFO [04-01|15:38:07.531] NAT mapped port                          proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 15:40:25 aviad-SEi geth[69608]: WARN [04-01|15:40:25.459] Beacon client online, but no consensus updates received in a while. Please fix your beacon client to follow the chain!
Apr 01 15:45:25 aviad-SEi geth[69608]: WARN [04-01|15:45:25.493] Beacon client online, but no consensus updates received in a while. Please fix your beacon client to follow the chain!
Apr 01 15:46:07 aviad-SEi geth[69608]: INFO [04-01|15:46:07.321] NAT mapped port                          proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 15:46:07 aviad-SEi geth[69608]: INFO [04-01|15:46:07.846] NAT mapped port                          proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 15:54:07 aviad-SEi geth[69608]: INFO [04-01|15:54:07.648] NAT mapped port                          proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 15:54:08 aviad-SEi geth[69608]: INFO [04-01|15:54:08.180] NAT mapped port                          proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 16:02:07 aviad-SEi geth[69608]: INFO [04-01|16:02:07.963] NAT mapped port                          proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 16:02:08 aviad-SEi geth[69608]: INFO [04-01|16:02:08.498] NAT mapped port                          proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 01 16:04:46 aviad-SEi geth[69608]: WARN [04-01|16:04:46.515] Served eth_coinbase                      conn=127.0.0.1:34248 reqid=3 duration="29.209µs" err="etherbase must be explicitly specified"

As you notice I've clear geth db (geth removedb) and now it still syncing, but without the 100% log.

[aviadshimoni]

Prysm syncing output, interval between them is ~4H:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   113  100   113    0     0   102k      0 --:--:-- --:--:-- --:--:--  110k
{
  "data": {
    "head_slot": "8761343",
    "sync_distance": "444",
    "is_syncing": true,
    "is_optimistic": true,
    "el_offline": false
  }
}
aviad@aviad-SEi:~$ curl http://localhost:3500/eth/v1/node/syncing | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   113  100   113    0     0   133k      0 --:--:-- --:--:-- --:--:--  110k
{
  "data": {
    "head_slot": "8762564",
    "sync_distance": "157",
    "is_syncing": true,
    "is_optimistic": true,
    "el_offline": false
  }
}

that means I need to wait till 15M blocks?

[aviadshimoni]

More logs:

Apr 01 18:04:56 aviad-SEi geth[69608]:   159: cumulative: 13870467 gas: 89057 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0x18454b157cefe67f9cdd5f7648ee238907aa1ebe6a058e4a206f2b8211fcd142 logs: [0xc0234d69a0] bloom: 00000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000008000000040000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000200000000000008000000000000000000000100004000000000800000000000000000000000001000044000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000 state:
Apr 01 18:04:56 aviad-SEi geth[69608]:   160: cumulative: 13891467 gas: 21000 contract: 0x0000000000000000000000000000000000000000 status: 1 tx: 0xb86918a2985a76599e2e9e820a5be95bd511070173c46aa51c880a4e097be515 logs: [] bloom: 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 state:
Apr 01 18:04:56 aviad-SEi geth[69608]: ##############################
Apr 01 18:04:56 aviad-SEi geth[69608]: ERROR[04-01|18:04:56.899] Beacon backfilling failed                err="retrieved hash chain is invalid: invalid gas used (remote: 13990223 local: 13891467)"
Apr 01 18:04:56 aviad-SEi geth[69608]: ERROR[04-01|18:04:56.899] Nil finalized block cannot evict old blobs
Apr 01 18:04:59 aviad-SEi geth[69608]: ERROR[04-01|18:04:59.138] Unhandled trie error in Trie.Prove       err="missing trie node 67120b27ad8a3045df012b9e4e2daae8ed047f5e52bc7800356b6b9cd633709c (path 0000010a060a0b) <nil>"
Apr 01 18:04:59 aviad-SEi geth[69608]: ERROR[04-01|18:04:59.141] Unhandled trie error in Trie.Prove       err="missing trie node 67120b27ad8a3045df012b9e4e2daae8ed047f5e52bc7800356b6b9cd633709c (path 0000010a060a0b) <nil>"
Apr 01 18:04:59 aviad-SEi geth[69608]: ERROR[04-01|18:04:59.142] State snapshotter failed to iterate trie err="missing trie node 67120b27ad8a3045df012b9e4e2daae8ed047f5e52bc7800356b6b9cd633709c (path 0000010a060a0b) <nil>"
Apr 01 18:05:00 aviad-SEi geth[69608]: WARN [04-01|18:05:00.933] Marked new chain head as invalid         hash=7213de..8e5ab6 badnumber=19,561,651 badhash=06fbbc..253d14
Apr 01 18:07:01 aviad-SEi geth[69608]: WARN [04-01|18:07:01.359] Beacon client online, but no consensus updates received in a while. Please fix your beacon client to follow the chain!

[rjl493456442]

What's the command you used for running Geth?

EDIT, you said

I'm using the guide from here:
https://github.com/metanull-operator/eth2-ubuntu/blob/master/v2/setup.md

[Unit]
Description=Go Ethereum Client
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=5
User=geth
WorkingDirectory=/home/geth
ExecStart=/usr/bin/geth --http --http.addr 0.0.0.0 --authrpc.jwtsecret=/srv/ethereum/secrets/jwtsecret --authrpc.vhosts="*"

[Install]
WantedBy=multi-user.target

Therefore the default hash scheme is selected.

[rjl493456442]

@aviadshimoni Can you post a bit more logs? A bad block log should also be printed.

[aviadshimoni]

@rjl493456442 is there an issue with the default hash schema?
adding more logs, I've executed geth removedb yesterday and kept the ancients.

aviad@aviad-SEi:~$ sudo journalctl -fu geth
Apr 02 12:12:23 aviad-SEi geth[122788]: INFO [04-02|12:12:23.110] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,566,532@9.08GiB bodies=19,566,471@535.19GiB receipts=19,566,471@184.32GiB eta=5m6.043s
Apr 02 12:12:31 aviad-SEi geth[122788]: INFO [04-02|12:12:31.115] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,566,532@9.08GiB bodies=19,566,471@535.19GiB receipts=19,566,471@184.32GiB eta=5m6.134s
Apr 02 12:12:39 aviad-SEi geth[122788]: INFO [04-02|12:12:39.120] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,566,532@9.08GiB bodies=19,566,471@535.19GiB receipts=19,566,471@184.32GiB eta=5m6.225s
Apr 02 12:12:47 aviad-SEi geth[122788]: INFO [04-02|12:12:47.125] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,566,532@9.08GiB bodies=19,566,471@535.19GiB receipts=19,566,471@184.32GiB eta=5m6.316s
Apr 02 12:12:50 aviad-SEi geth[122788]: INFO [04-02|12:12:50.651] NAT mapped port                          proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 02 12:12:51 aviad-SEi geth[122788]: INFO [04-02|12:12:51.230] NAT mapped port                          proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 02 12:12:55 aviad-SEi geth[122788]: INFO [04-02|12:12:55.129] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,566,532@9.08GiB bodies=19,566,471@535.19GiB receipts=19,566,471@184.32GiB eta=5m6.408s
Apr 02 12:13:03 aviad-SEi geth[122788]: INFO [04-02|12:13:03.133] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,566,532@9.08GiB bodies=19,566,471@535.19GiB receipts=19,566,471@184.32GiB eta=5m6.499s
Apr 02 12:13:11 aviad-SEi geth[122788]: INFO [04-02|12:13:11.137] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,566,532@9.08GiB bodies=19,566,471@535.19GiB receipts=19,566,471@184.32GiB eta=5m6.590s
Apr 02 12:13:19 aviad-SEi geth[122788]: INFO [04-02|12:13:19.141] Syncing: chain download in progress      synced=100.00% chain=728.59GiB headers=19,566,532@9.08GiB bodies=19,566,471@535.19GiB receipts=19,566,471@184.32GiB eta=5m6.681s

{
  currentBlock: 19566471,
  healedBytecodeBytes: 1195891,
  healedBytecodes: 162,
  healedTrienodeBytes: 788288073,
  healedTrienodes: 2721698,
  healingBytecode: 0,
  healingTrienodes: 1,
  highestBlock: 19566490,
  startingBlock: 19564802,
  syncedAccountBytes: 59795785019,
  syncedAccounts: 242883123,
  syncedBytecodeBytes: 8218954202,
  syncedBytecodes: 1180813,
  syncedStorage: 1099158418,
  syncedStorageBytes: 237049600581,
  txIndexFinishedBlocks: 2350000,
  txIndexRemainingBlocks: 0
}

aviad@aviad-SEi:~$ curl http://localhost:3500/eth/v1/node/syncing | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   113  100   113    0     0   146k      0 --:--:-- --:--:-- --:--:--  110k
{
  "data": {
    "head_slot": "8768141",
    "sync_distance": "629",
    "is_syncing": true,
    "is_optimistic": true,
    "el_offline": false
  }
}

I believe I'm in the last phase where geth verifies ever block, I wonder how long it would take.

BTW, thank you for responding @rjl493456442

[rjl493456442]

Your state is somehow corrupted by missing a trie node.

You might be able to fix it by executing these scripts:

• Stop the geth node gracefully
• Execute

db put --datadir <your data dir> 0x67120b27ad8a3045df012b9e4e2daae8ed047f5e52bc7800356b6b9cd633709c 0xf8669d35fce6efd2272cdb9ea2eacce639fab06c112ac8be435a182bae332defb846f8440180a0bbd13d463ce2111713bb0d6ad117b0ba991bc32336d501e872703c4ad6bc3c90a0905982d3bc98b3ad75d5ca7a92446cc8923fae39a2352245000f9d3e9dc0f3ee

Let's see if the state can be fixed.

I will keep investigating the root issue of it.

[aviadshimoni]

Kudos for the quick response, again!

INFO [04-02|14:19:02.704] Maximum peer count                       ETH=50 total=50
INFO [04-02|14:19:02.705] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [04-02|14:19:02.707] Set global gas cap                       cap=50,000,000
INFO [04-02|14:19:02.708] Initializing the KZG library             backend=gokzg
INFO [04-02|14:19:02.879] Using pebble as the backing database
INFO [04-02|14:19:02.879] Allocated cache and file handles         database=/home/geth/.ethereum/geth/chaindata cache=512.00MiB handles=524,288
INFO [04-02|14:19:09.658] Opened ancient database                  database=/home/geth/.ethereum/geth/chaindata/ancient/chain readonly=false
Previous value: 0xf8669d35fce6efd2272cdb9ea2eacce639fab06c112ac8be435a182bae332defb846f8440180a0bbd13d463ce2111713bb0d6ad117b0ba991bc32336d501e872703c4ad6bc3c90a0905982d3bc98b3ad75d5ca7a92446cc8923fae39a2352245000f9d3e9dc0f3ee

Logs after starting geth immediately:
`

eth.syncing
false
exit
aviad@aviad-SEi:~$ sudo journalctl -fu geth
Apr 02 14:20:02 aviad-SEi geth[127649]: INFO [04-02|14:20:02.286] NAT mapped port proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 02 14:20:03 aviad-SEi geth[127649]: INFO [04-02|14:20:03.016] NAT mapped port proto=TCP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 02 14:20:03 aviad-SEi geth[127649]: INFO [04-02|14:20:03.557] NAT mapped port proto=UDP extport=30303 intport=30303 interface="UPNP IGDv1-PPP1"
Apr 02 14:20:09 aviad-SEi geth[127649]: INFO [04-02|14:20:09.318] Looking for peers peercount=3 tried=163 static=0
Apr 02 14:20:19 aviad-SEi geth[127649]: INFO [04-02|14:20:19.991] Looking for peers peercount=4 tried=132 static=0
Apr 02 14:20:30 aviad-SEi geth[127649]: INFO [04-02|14:20:30.056] Looking for peers peercount=4 tried=1 static=0
Apr 02 14:20:34 aviad-SEi geth[127649]: WARN [04-02|14:20:34.239] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
Apr 02 14:20:38 aviad-SEi geth[127649]: WARN [04-02|14:20:38.554] Served eth_coinbase conn=127.0.0.1:56772 reqid=3 duration="23.02µs" err="etherbase must be explicitly specified"
Apr 02 14:20:40 aviad-SEi geth[127649]: INFO [04-02|14:20:40.513] Looking for peers peercount=4 tried=42 static=0
Apr 02 14:20:50 aviad-SEi geth[127649]: INFO [04-02|14:20:50.578] Looking for peers peercount=4 tried=53 static=0

`

[rjl493456442]

Previous value: 0xf8669d35fce6efd2272cdb9ea2eacce639fab06c112ac8be435a182bae332defb846f8440180a0bbd13d463ce2111713bb0d6ad117b0ba991bc32336d501e872703c4ad6bc3c90a0905982d3bc98b3ad75d5ca7a92446cc8923fae39a2352245000f9d3e9dc0f3ee

Wait! You do have this entry in your database. Did you apply the script twice?

OK, you said I've executed geth removedb yesterday and kept the ancients.. You just need to wait until the state is fully synced.

[aviadshimoni]

No, probably because I've executed geth removedb yesterday, so entry now exist.

I don't see any special logs from geth now,
but beacon responding:

pr 02 15:12:43 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:43" level=debug msg="Could not decode stream message" error="i/o deadline reached" prefix=sync topic="/eth2/beacon_chain/req/ping/1/ssz_snappy"
Apr 02 15:12:45 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:45" level=error msg="Could not compute head from new attestations" error="0x60f0863d3241c14a564f8781caed4d1a2df4ad20be940cfdf8ff88c886fc223e: unknown justified root" prefix=blockchain
Apr 02 15:12:45 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:45" level=debug msg="Peer disconnected" activePeers=68 multiAddr="/ip4/108.51.197.38/tcp/13000/p2p/16Uiu2HAmBbgJTf4DoqJ2Syo8GFcCRYnrkhUDtq2QTcq6nMVz37ZE" prefix=p2p
Apr 02 15:12:47 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:47" level=error msg="Could not compute head from new attestations" error="0x60f0863d3241c14a564f8781caed4d1a2df4ad20be940cfdf8ff88c886fc223e: unknown justified root" prefix=blockchain
Apr 02 15:12:49 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:49" level=debug msg="Latest eth1 chain event" blockHash=0x5f951778663f244471ecc33e254e7ff500d8074b6d286505f8aa7940aeb4bd69 blockNumber=19566471 prefix=execution
Apr 02 15:12:49 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:49" level=warning msg="Execution client is not syncing" prefix=execution
Apr 02 15:12:50 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:50" level=debug msg="Peer disconnected" activePeers=67 multiAddr="/ip4/38.107.226.72/tcp/13000/p2p/16Uiu2HAm6EKXMSgykDVhnGCHAS43NjJfB5oCaFmJuHhn2vDuxLuG" prefix=p2p
Apr 02 15:12:51 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:51" level=debug msg="Could not decode stream message" error="i/o deadline reached" prefix=sync topic="/eth2/beacon_chain/req/ping/1/ssz_snappy"
Apr 02 15:12:54 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:54" level=debug msg="Aggregated unaggregated attestations" duration=8.532094ms prefix="pool/attestations"
Apr 02 15:12:54 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:54" level=info msg="Peer summary" activePeers=67 inbound=0 outbound=67 prefix=p2p
Apr 02 15:12:57 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:57" level=error msg="Could not compute head from new attestations" error="0x60f0863d3241c14a564f8781caed4d1a2df4ad20be940cfdf8ff88c886fc223e: unknown justified root" prefix=blockchain
Apr 02 15:12:57 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:57" level=debug msg="Peer connected" activePeers=71 direction=Outbound multiAddr="/ip4/89.160.100.168/tcp/9000/p2p/16Uiu2HAmQm2n572nb3eCCL6wrTifTxKUv4FNQLZK3yE4w4AZAk9u" prefix=p2p
Apr 02 15:12:59 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:12:59" level=error msg="Could not compute head from new attestations" error="0x60f0863d3241c14a564f8781caed4d1a2df4ad20be940cfdf8ff88c886fc223e: unknown justified root" prefix=blockchain
Apr 02 15:13:00 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:13:00" level=debug msg="Requesting blobs" capacity=64 count=62 peer=16Uiu2HAmBArzrugMTdvX8CXyTPYVrrWphURtxyGc8gSgKas8P1qk prefix=initial-sync score="[0.0%, raw: 0.00,  blocks: 0/640]" start=8768144
Apr 02 15:13:00 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:13:00" level=debug msg="Sending blob by range request" count=62 prefix=sync startSlot=8768144 topic="/eth2/beacon_chain/req/blob_sidecars_by_range/1"
Apr 02 15:13:03 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:13:03" level=debug msg="Latest eth1 chain event" blockHash=0x5f951778663f244471ecc33e254e7ff500d8074b6d286505f8aa7940aeb4bd69 blockNumber=19566471 prefix=execution
Apr 02 15:13:03 aviad-SEi prysm.sh[69571]: time="2024-04-02 15:13:03" level=warning msg="Execution client is not syncing" prefix=execution