Handle disconnect/reconnect error display/status from JS API better

[nbauernfeind]

As a user of the system, it would be nice to know when my worker crashes, possibly why it crashed, and how long ago it appears to have crashed.

The worker can, on best effort, attempt to log to the user the error that occurred just prior to it exiting. This error would hit the console log.

However, if enough streams/rpcs fail then the UI drops into a "everything is broken" mode, which clears the page/layout. It would be nice if we can alert the user that things are not working from the Javascript Client's perspective, without clearing this important console output.

I suggest that there might be a solution in the jsapi firing an event that causes the UI to either:

• create the modal-pop-up with the text in the event
• update the existing modal-pop-up
• as well as be able to close the pop-up if it truly was a transient, temporary, disconnect from the backend

I like Google's approach to this:

Perhaps the event can have a "message", a "start-time", and boolean flag to say where or not the "click-to-refresh" link is shown.

[mofojed]

@dsmmcken for input. We can/should improve the disconnected/reconnecting phase, to show a better error and possibly an action to take, but I'm not sure showing a toast is the best option. We'd need to at least disable anything that tries to communicate with the server, which is pretty much every panel.

Maybe we just have a button on the error screen to view logs or to download logs or something.

[rcaudy]

I like the idea of a yellow (or red) banner, more than a toast. I feel like toasts tend to be transient.

[mofojed]

Digging into this a bit... Reproducing using the example in deephaven/deephaven-core#888

First the Table in the UI gets an error, "all subConns are in TransientFailure". UI blanks out the table with the error at this point. I do not see an error in the console/logs tab otherwise though. Then 10s later, I get the Connection Unavailable error, which blanks out the whole UI.

@nbauernfeind:

1. For a Connection Unavailable error, is it expected to be recoverable? Currently the UI treats it as disconnected, didn't think reconnect implemented yet. @niloc132 may have something to note as well.
2. Does the table failure "all subConns are in TransientFailure" always lead to this? Should that automatically trigger a connection failure immediately instead of waiting 10s?

[rcaudy]

"all subConns are in TransientFailure"

Hardly seems like a user-friendly error message.
I'd also like to have something persistent in the UI revealing the state.

[nbauernfeind]

Transient failures are not always caused by the worker exiting unexpectedly. However, when the worker exits unexpectedly you can expect to receive transient failures until the worker is restarted (i.e. they are not actually transient). It is possible to receive a transient failure when everything was otherwise working properly and could be recovered from gracefully.

Note that this failure message is coming from envoy who is trying to say that there are no nodes in the gRPC pool. We use singleton-node pools. This message is going to be sent if 1) the worker is still starting; 2) the worker has exited and envoy is waiting for it to start again; and 3) if envoy and the gRPC server hiccup temporarily (this is unlikely scenario when they run on the same machine, and still unlikely in a local network situation -- but is nonetheless possible in both).