TimeoutException after 0:00:15.000000: Pool deadlock: all resources have been allocated for too long.

[davidmartos96]

Hello!
On our low traffic staging environment we started to notice that the postgres will hang after a while. The thrown error is a Timeout error from the pool package

https://github.com/dart-lang/pool/blob/bf27900420ba382b6e5c0484ab3c79daad703dcd/lib/pool.dart#L329

I'm assuming that the reason is the following timeouts. Does the settings.connectTimeout match the Pool API timeout intentions? Maybe they mean different things here. Pool mentions that it will throw if there is no activity, which might not be what we want in the postgres side.

/// If [timeout] is passed, then if that much time passes without any activity
/// all pending [request] futures will throw a [TimeoutException]. This is
/// intended to avoid deadlocks.

postgres version 3.4.8

[isoos]

On a cursory look, it may be some connection instances may be stuck? Could you please give me more details about the setup and how you are using them?

[davidmartos96]

The main issue is that it doesn't seem to be a recoverable exception.
Our pool instance is created once in the server process. If this is thrown, requests to the pool will hang indefinitely, requiring a server restart.

Other possible errors in the pool are correctly handled by the postgres pool logic (network, server...), so the single instance pool object has been working great for us except for this particular error.

[davidmartos96]

@isoos thank you. I was just writing additional context. Hope that helps.

[davidmartos96]

For this particular staging setup we have a Cloud run instance for the API (US) that creates a Pool with a postgres hosted in a docker container in Hetzner (Germany).

[davidmartos96]

Also, redeploying the Cloud Run API makes it work again, no need to restart the Postgres. That's why we suspect being an issue in the library/API server

[isoos]

@davidmartos96: do you happen to know if this is the connection lock or the generic pool's timeout that is being triggered?

I think we should rewrite the affected parts using the package:pool with timeout anyway...

[davidmartos96]

Unfortunately I don't. The stacktrace doesn't include that information becase the error is thrown from an async context.

[isoos]

@davidmartos96: I did an alternative implementation of that timeout, will land and publish it soon.

[davidmartos96]

@isoos That's great to hear, thank you!

Out of curiosity, did you manage to reproduce the issue (or force it to occur) to then observe the pool being stuck/frozen?
I tried to but I wasn't able to trigger those timeouts locally.