Keyper hanging on websocket disconnect

[ylembachar]

We identified a few places in which goroutines might block without listening for context cancellations:

• slotticker.go: we don't check for cancellations when waiting for the timer (

  rolling-shutter/rolling-shutter/medley/slotticker/slotticker.go

  Line 95 in 94d52a9

  <-timer.C

  ). At most, this should block for the slot time though.
• channel.go: There's a function that forwards from one channel to the other. We block when the outgoing channel is full without checking for cancellations (

  rolling-shutter/rolling-shutter/medley/channel/channel.go

  Line 15 in 94d52a9

  send <- val

  ). So far, all channels we pass seem to be unbounded, but still something we should fix just to be sure.
• eonkeypublisher.go: blocks without checking for cancellations (

  rolling-shutter/rolling-shutter/eonkeypublisher/eonkeypublisher.go

  Line 79 in 94d52a9

  p.keys <- key

  ), but channel has size 32 which we shouldn't have hit yet.
• retry.go: haven't fully understood the code yet and I'm unsure if it's critical or not, but it does seem like we should check for cancellations here. At the very least, we might block here for the max retry interval of 60s (

  rolling-shutter/rolling-shutter/medley/retry/retry.go

  Line 69 in 94d52a9

  iter <- <-r.clock.After(nextTick)

  ):

At some places in which we send something to a channel, it seems we might block, but since the involved channels are unbounded it should not cause any issues (still cleaner to fix in my opinion):

• channelNew... functions in gnosis and service keyper.go (e.g.,

  rolling-shutter/rolling-shutter/keyperimpl/gnosis/keyper.go

  Line 299 in 94d52a9

  kpr.newBlocks <- ev

  )
• newslot.go: kpr.decryptionTriggerChannel (

  rolling-shutter/rolling-shutter/keyperimpl/gnosis/newslot.go

  Line 269 in 94d52a9

  kpr.decryptionTriggerChannel <- event

  )

Some parts of unused codes are also susceptible:

rolling-shutter/rolling-shutter/shdb/notify.go

Line 25 in 94d52a9

chann <- notification

(blocking channel send),

rolling-shutter/rolling-shutter/medley/broker.go

Line 37 in 94d52a9

func StartNewBroker[T any](nonBlocking bool) *Broker[T] {

(goroutine without cancel context)

[blockchainluffy]

Summary of Modified and Skipped Changes:

Modified

1. slotticker.go: The existing timer.C read could block until the next slot time, even after context cancellation. It’s better to cancel the process immediately instead of waiting for the full slot duration.
2. channel.go: The function forwards messages from one channel to another. If the receiving channel is full, the send operation can block indefinitely and ignore context cancellation.
   3. keyper.go (gnosis & shutterservice) and newslot.go: These use unbuffered channels. In such cases, a send blocks until a receiver is available, which can prevent the goroutine from exiting cleanly when the context is cancelled.

Skipped

1. eonkeypublisher.go: The channel has a buffer size of 32, and since it’s used for eon keys (which are published infrequently), it’s unlikely to fill up or cause blocking.
2. retry.go: The retry iterator is created and used within the same function, without spawning additional goroutines. It cooperates naturally and won’t block external routines.
3. notify.go and broker.go: These parts of the codebase are currently unused.