Peer ID doesn't match libp2p public key

[dapplion]

Describe the bug

Running in denver gnosis testnet, Lodestar is unable to discover peers because all discovered peers throw

Aug-01 23:17:38.392[NETWORK]         debug: Error dialing discovered peer peer=16...zfqTuk Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Peer ID doesn't match libp2p public key.
AggregateError: Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Peer ID doesn't match libp2p public key.
    at maybeSettle (/usr/app/node_modules/p-some/index.js:31:11)
    at /usr/app/node_modules/p-some/index.js:69:23
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
Aug-01 23:17:38.393[NETWORK]         debug: Error dialing discovered peer peer=16...zfqTuk Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Peer ID doesn't match libp2p public key
AggregateError: Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Peer ID doesn't match libp2p public key.
    at maybeSettle (/usr/app/node_modules/p-some/index.js:31:11)
    at /usr/app/node_modules/p-some/index.js:69:23
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
Aug-01 23:17:38.393[NETWORK]         debug: Error dialing discovered peer peer=16...JgcSVS Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Peer ID doesn't match libp2p public key.
AggregateError: Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Peer ID doesn't match libp2p public key.
    at maybeSettle (/usr/app/node_modules/p-some/index.js:31:11)
    at /usr/app/node_modules/p-some/index.js:69:23
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
Aug-01 23:17:38.393[NETWORK]         debug: Error dialing discovered peer peer=16...JgcSVS Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Peer ID doesn't match libp2p public key
AggregateError: Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Peer ID doesn't match libp2p public key.
    at maybeSettle (/usr/app/node_modules/p-some/index.js:31:11)
    at /usr/app/node_modules/p-some/index.js:69:23
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
Aug-01 23:17:38.394[NETWORK]         debug: Error dialing discovered peer peer=16...yteSaC Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Peer ID doesn't match libp2p public key.

Expected behavior

Not reject all peers

[wemeetagain]

Related comment #4243 (comment)

Got a better stack trace there.
Seems we initialize a noise handshake w a peer id (optionally), and the peer id is also exchanged during the handshake.

[dapplion]

Closing for now since this issue is not actionable. To collect more data on future cases:

• This PR will tell us the mismatched key fix: Add peer id str to mismatch error js-libp2p-noise#273
• By enabling metrics we can track the frequency of this type of errors

@wemeetagain none of this metrics show up in our infra. How can noise metrics be enabled? https://github.com/ChainSafe/js-libp2p-noise/blob/master/src/metrics.ts