Connection is not shutting down properly, which makes reconnects within ~10 seconds impossible

[marcus-pousette]

I noticed an issue when I have to nodes in two browser tabs, and I refresh on of the tabs. The connection is able to resume immediately, but

1. After a few seconds the WebRTC connection closes in the tab that did not refresh (even though the client in both tabs are live and connected)
2. The tab that refreshes still thinks the connection is live which makes messages not delivering

To reproduce the issues partially:

Put a log here
https://github.com/libp2p/js-libp2p-webrtc/blob/efe5ce70bdffdb67c237cfb4688afd562528a050/src/maconn.ts#LL68C9-L68C9

and run the browser-to-browser example.

Node 1 connects to the relay, and Node 2 connects to Node 1 relayed address.

After connection, close Node 2 and observe that the state change does not appear immediately in the logs.

--

If Node 2 would restart with the same PeerId and connects to Node 1 before the state change has happened, Node 1 will terminate Node 2 wrongfully as

https://github.com/libp2p/js-libp2p-webrtc/blob/efe5ce70bdffdb67c237cfb4688afd562528a050/src/maconn.ts#L70

will bubble up.

Since the browser-to-browser example is using new PeerIds each time the issue becomes less severe, but in real world scenarios where you run nodes with the same PeerId you will have nodes running with broken connections, as one side of the connection has wrongfully terminated it due to the reason outlined above.

[mxinden]

The tab that refreshes still thinks the connection is live which makes messages not delivering

I am assuming that the underlying WebRTC connection is no longer alive at this point, correct? In other words, while the js-libp2p state might think it is still alive, the underlying browser stack does not have a reference to the connection anymore.

[marcus-pousette]

I am assuming that the underlying WebRTC connection is no longer alive at this point, correct? In other words, while the js-libp2p state might think it is still alive, the underlying browser stack does not have a reference to the connection anymore.

yeap. Though I don't observe any onconnectionstatechange -> disconnected on the tab that I did refresh. Only in the other tab. I am not sure what this means in practice.

[maschad]

If Node 2 would restart with the same PeerId and connects to Node 1 before the state change has happened, Node 1 will terminate Node 2 wrongfully as

libp2p/js-libp2p-webrtc@efe5ce7/src/maconn.ts#L70

will bubble up.

I am not sure if this is still an issue as should the redial occur from the auto-dialler (given the previous connection was closed) then a new connection will be created, so it would have been correctly terminated given that Node B was in fact shutdown.

Please re-open if you feel this is still an issue using our issue template

[marcus-pousette]

Even though there would be an auto-dialer that re-establishes (even though that was not anything I did observe) the connection then again needs to be re-established. In my use-case if I am doing chatting over webrtc, or video-streaming, the messages will not be delivered for a while, because of this. I can not use libp2p webrtc for any of my project because this issue right now.

I can perhaps try to reproduce it again, but relying on the auto-dialer feels like a last resort, when the connection just can be closed properly and this issue will not be present in the first place.

[maschad]

Perhaps I am misunderstanding but following your reproduction instructions:

To reproduce the issues partially:

Put a log here libp2p/js-libp2p-webrtc@efe5ce7/src/maconn.ts#LL68C9-L68C9

and run the browser-to-browser example.

Node 1 connects to the relay, and Node 2 connects to Node 1 relayed address.

After connection, close Node 2 and observe that the state change does not appear immediately in the logs.

--

If Node 2 would restart with the same PeerId and connects to Node 1 before the state change has happened, Node 1 will terminate Node 2 wrongfully as

libp2p/js-libp2p-webrtc@efe5ce7/src/maconn.ts#L70

will bubble up.

If Node 2 was closed as mentioned, then it would be correct for Node 1 to close that connection and subsequently when it re-discovers Node 2 after it is restarted, creates a new connection via the auto-dialler. I am not sure what you mean by the connection isn't closed properly then. What was your expected behaviour in this scenario?

[marcus-pousette]

Yeap that behaviour is alright, but the problem is that Node 1 does not realize that Node 2 has disconnected, or the webrtc datachannel does not know that at least until a few seconds has passed. Now, due to this, if Node 2 not restablishes the connection (using the auto-dialer) within a few seconds ( < 10 seconds ), prior to that Node 1 has realized that Node 2 has disconnected, Node 2 is going to rely/re-establish on a datachannel that will in a moment close.

If Node 2 reconnects after 10 seconds, this issue is not present, because the datachannel that was used, has time to close, and the new connection will use a "fresh" data channel (?). Whereas if you re-connect to quickly you are going to rely on a channel that will eventually close.

Reconnection are quite common, for example, when you refresh you browser tab to view different content, or when you are working with the frontend using hot-reloading. See this for others having related problems: https://filecoinproject.slack.com/archives/C03L0G3B4RX/p1686756871439659

[marcus-pousette]

If you use chrome devtools and see the live thewebrtc connections while you close on of the tabs in the browser-to-browser example, you can see that there is a significant delay until the datachannel gets removed.

[maschad]

Yeap that behaviour is alright, but the problem is that Node 1 does not realize that Node 2 has disconnected, or the webrtc datachannel does not know that at least until a few seconds has passed. Now, due to this, if Node 2 not restablishes the connection (using the auto-dialer) within a few seconds ( < 10 seconds ), prior to that Node 1 has realized that Node 2 has disconnected, Node 2 is going to rely/re-establish on a datachannel that will in a moment close.

If Node 2 reconnects after 10 seconds, this issue is not present, because the datachannel that was used, has time to close, and the new connection will use a "fresh" data channel (?). Whereas if you re-connect to quickly you are going to rely on a channel that will eventually close.

Reconnection are quite common, for example, when you refresh you browser tab to view different content, or when you are working with the frontend using hot-reloading. See this for others having related problems: https://filecoinproject.slack.com/archives/C03L0G3B4RX/p1686756871439659

Thanks for clarifying. Based on that description it seems to be related to how we are closing the RTCDataChannel . I am re-opening and will investigate.

[marcus-pousette]

Thank you! Ye I think so. I tried to debug a bit, but there are many moving parts to dive into.

A wild guess, one thing I thought of was that if there perhaps is an id associated with RTCDataChannel, and if the id the is the same for each session, the client perhaps might re-use the"old" RTCDataChannel in the next session, I solution would be to generate new id for every session.