P2P: Failed to process network dispatch: SendError

[CharlieC3]

Describe the bug
A stacks-node (2.05.0.2.2) emitting events to a stacks-blockchain API (4.0.3) crashed at STX block 43233, printing the following:

Process abort due to thread panic: panicked at 'explicit panic', testnet/stacks-node/src/neon_node.rs:779:29
Failed to write to fd: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }
Failed to send data to socket TcpStream { addr: 172.25.26.7:59810, fd: 96 }: SendError("Failed to send 258 bytes")
http:id=236,request=true,peer=52.0.54.100:20443: failed to send on HTTP conversation: SendError("Failed to send 258 bytes")
P2P: Failed to process network dispatch: SendError("Failed to send 258 bytes")
Panic backtrace:    0: stacks_node::main::{{closure}}
             at src/testnet/stacks-node/src/main.rs:59:18
   1: std::panicking::rust_panic_with_hook
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:702:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:586:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/sys_common/backtrace.rs:138:18
   4: rust_begin_unwind
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
   5: core::panicking::panic_fmt
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
   6: core::panicking::panic
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:48:5
   7: stacks_node::neon_node::spawn_peer::{{closure}}
             at src/testnet/stacks-node/src/neon_node.rs:779:29
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/sys_common/backtrace.rs:122:18
   8: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/thread/mod.rs:498:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:492:40
      std::panicking::try
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:456:19
      std::panic::catch_unwind
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panic.rs:137:14
      std::thread::Builder::spawn_unchecked_::{{closure}}
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/thread/mod.rs:497:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/function.rs:227:5
   9: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/alloc/src/boxed.rs:1861:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/alloc/src/boxed.rs:1861:9
      std::sys::unix::thread::Thread::new::thread_start
             at rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/sys/unix/thread.rs:108:17

Expected behavior
I would've expected the stacks-node to retry the failed connection indefinitely instead of crashing.

Environment (please complete the following information):
blockstack/stacks-blockchain:2.05.0.2.2 Docker image

[kantai]

Yeah -- this looks like an error that should be handled by the network dispatch, and not propagated up into neon_node (where, if the node is event-driven, it will panic).

The last log message appears to be here:

https://github.com/stacks-network/stacks-blockchain/blob/master/src/net/connection.rs#L1130

I'm not sure where in the network dispatch this is getting invoked, though. I think @jcnelson should have more insight.

[jcnelson]

Can you post the last 1000 log lines prior to this? I need more context to figure out exactly where this is happening.

The network dispatch code should definitely be absorbing all of these kinds of errors, so finding the one place where it does not is going to be a challenge without some additional context.

[jcnelson]

Any update on this? Do you have more log data? Kinda want to ship this in 2.05.0.3.0.

[CharlieC3]

@jcnelson Here you go!

mainnet-stacks-node-follower-crash.csv

[jcnelson]

Pushed a fix (#3228). The problem was that we were accidentally propagating an error from the mempool sync sub-state-machine in the p2p state machine. I went ahead and made all sub-state-machines, as well as PeerNetwork::dispatch_network() infallible so this can never happen again.

[jcnelson]

It took all of 30 seconds to figure this out with the logs. Thanks for supplying them!

[jcnelson]

Should be fixed by #3232. Please re-open if not.

[blockstack-devops]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.