Understand when disconnections are graceful or not

[SteveSandersonMS]

I'm splitting this out of comments in #12788 so we track dealing with this specifically. Since implementing the graceful disconnect support, we don't know in which real-world scenarios reconnection will be attempted or not.

Relevant comments:

---

@SteveSandersonMS:

Is the event of an ungraceful disconnection common?

I'd go further and check we can prove it's even possible in the real world, not just tests.

• If you deploy an app to Azure, connect through wifi (e.g., on a phone), then walk out of range of the wifi signal, does that cause a graceful or a disgraceful disconnect?
• Which one happens if you switch the phone into flight mode?
• Which one happens if you're on a laptop and close the lid?

We ought to be sure that disgraceful disconnects are the common case (except when genuinely closing the browser/tab), otherwise we've invested a lot into a feature that will never be used.

---

@SteveSandersonMS
There is a different technical solution here, which is to only do a "graceful disconnect" when we receive the beacon from an unload handler. That bypasses all the uncertainty about whether the network traffic is right about whether the user really intended to disconnect.

Drawbacks to using beacon:

• Doesn't cover the case where the browser crashes (we'd keep the circuit open, even though there's actually no hope of reconnecting)
• Doesn't work if you lack client-server affinity, since the beacon could go to any server
• Makes it much easier to fill the server with abandoned circuits, since you can just choose not to send the beacon. The existing design is harder to trick because you need to leave a half-open connection, which involves dropping down to a very low level in the network stack, beyond what most networking APIs allow.

---

@javiercn

• Doesn't work if you lack client-server affinity, since the beacon could go to any server

Reconnection wouldn't work either.

• Makes it much easier to fill the server with abandoned circuits, since you can just choose not to send the beacon. The existing design is harder to trick because you need to leave a half-open connection, which involves dropping down to a very low level in the network stack, beyond what most networking APIs allow.

Re-connection is best effort and we can always remove the circuits in a FIFO fashion.

• Doesn't cover the case where the browser crashes (we'd keep the circuit open, even though there's actually no hope of reconnecting)

If the browser crashes I think it would be treated as a graceful disconnect in the current scenario.

I'd go further and check we can prove it's even possible in the real world, not just tests.

• If you deploy an app to Azure, connect through wifi (e.g., on a phone), then walk out of range of the wifi signal, does that cause a graceful or a disgraceful disconnect?
• Which one happens if you switch the phone into flight mode?
• Which one happens if you're on a laptop and close the lid?

We ought to be sure that disgraceful disconnects are the common case (except when genuinely closing the browser/tab), otherwise we've invested a lot into a feature that will never be used.

I still think there are ways we can cause it to be ungracefully terminated, the main thing is that it won't be treated as ungraceful unless you are in the middle of receiving a message.

• For example, if we throw while waiting for a message to arrive, it won't be considered ungraceful, and that might happen either due to a network loss or the browser being closed.
• If we throw after we've read the header but not the entire payload, that can cause the server to barf too.

I plan to physically try some of these things:

• Turn of phone wifi.
• Turn on airplane mode.
• Go away from wifi range.
• Kill browser process.

Update:
I managed to make this work in test scenarios. It involves a bit of trickery, but it works. We provide our own IHttpWebSocketsFeature, it does the same thing that the WebSocketsMiddleware one does.

• We wrap the websocket it creates with a test socket of our own we can use to trigger exceptions whenever we want.
• We do this conditionally on tests based on the presence of a cookie.
• We trigger the crash with a request from javascript to a special controller.

Although we can make sure the feature works, we need to better understand the scenarios E2E to make sure there is no situation in which it can have issues where a temporary disconnect gets interpreted as a graceful one.

[SteveSandersonMS]

I think the summary of the current state is:

• @javiercn has found a way to make the E2E tests run and pass again ([Blazor][Fixes #12788] Fixes blazor reconnection tests #12813)
• But we don't know whether that reflects any real-world scenario
• From comments above, sounds like @javiercn leans towards a beacon-based system for graceful disconnect as it's less ambiguous. I would do too. However that's another nontrivial work item if we choose to do it.
• We should actually validate what the current behavior is in the cases described (wifi signal dropout, lock phone, put phone in flight mode, suspend laptop). If all of them behave properly (treat as ungraceful disconnect) then we're probably OK already. If they falsely treat it as a graceful disconnect, our current feature is not useful and needs to be replaced with something like the beacon-based solution.

[BrennanConroy]

I hosted a SignalR app and captured the exception from OnDisconnectedAsync to see if graceful vs. ungraceful could be detected reliably.

I tested connecting via WiFi and turning the WiFi off, connection via WiFi and going to Airplane Mode, connection via browser and killing the process. All 3 resulted in no exceptions because the server notices a missed Ping within the timeout window and kills the connection without triggering an error. The connection isn't invalid as far as the server (or OS) is concerned when the client side disappears.

[rynowak]

@javiercn - @SteveSandersonMS - I guess that's the answer then 😆

We should implement this with a beacon in the future I spose.

[SteveSandersonMS]

Does ”no exception” mean that Blazor sees it as a graceful disconnect and hence discards the circuit?

If so, that basically means our entire reconnection feature is currently completely useless, right? Is it really that bad?

[rynowak]

The impression that I get is that there are three cases:

• IO error happens (triggered by a protocol violation or incomplete payload, etc)
• Client simply goes away without explicitly saying goodbye (missed heartbeat)
• Client hangs up gracefully

The first case is correctly reported as an exception.

The third case is correctly reported as a non-exception.

The second case isn't reported as an exception - which is the kind of thing you'd encounter by turning on airplane mode. It sounds like we remove the feature and try again in the future with a different approach.

[javiercn]

I think this is covered by #12853

[javiercn]

In our case we are simply going to disconnect.

[rynowak]

Closing since this is done. @BrennanConroy - clap back if you disagree.