Retry on "Broadcaster is dead"

[anatolystansler]

Hello! Thanks for an awesome project!

Problem

If my Postgres restarts postgres-websockets service stops receiving events and logs "Broadcaster is dead". There are no connection retry attempts as far as I can see. Websocket connection can still be established, but the client does not receive any events accordingly.

Logs

2020-05-27 23:57:28.064 MSK postgres-websockets 0.6.0.0 / Connects websockets to PostgreSQL asynchronous notifications.
2020-05-27 23:57:28.068 MSK Listening on port 9509
2020-05-27 23:57:53.906 MSK ConnectionClosed
2020-05-27 23:58:52.427 MSK CloseRequest 1000
2020-05-27 23:59:54.351 MSK FATAL: terminating connection due to administrator command
2020-05-27 23:59:54.351 MSK Broadcaster is dead
2020-05-28 00:11:29.789 MSK CloseRequest 1001
2020-05-28 00:11:37.310 MSK CloseRequest 1001

Expectations

I would expect postgres-websockets service to retry to Postgres after the connection is lost. Is there such a possibility? Is it configurable?

[diogob]

@anatolystansler this feature is not implemented. But I agree that it's a must-have.

[anatolystansler]

Do you have plans to implement it? 🙂

[diogob]

I will take a stab at it after finishing #54 , I can't give you an ETA since I'm working only on my free time at the moment.

If you are willing to put some time in a PR I would be glad to help you navigate the needed code changes.

[anatolystansler]

Gotcha, thanks!

As a fast solution it would be good to convert the error log to fatal exit. With this change it would be possible to restart service with external tools, e.g. k8s.

[diogob]

@anatolystansler that is a great idea, I'll roll out that change before trying to restore the connection state.

[diogob]

@anatolystansler I have opened #57 but I had done very little testing. Would you be able to compile from that branch and confirm that it fits your use case?
I'm also not sure if the server is properly cleaning all resources (since most is done by the warp library) so if you see any lingering network socket after the shutdown please let me know.

[diogob]

Released in 0.7.0.0 to hackage and dockerhub. I'll leave this open since I want to improve on this and perhaps have the service reopen the connection on its own.

[anatolystansler]

@diogob thanks for the fix, works as intended :)

[diogob]

Actually after some thought, I believe that this is better handled outside of postgres-websockets. The additional code complexity to create a fully-self healing process is more than what I'm willing to handle on a one-developer volunteer project.