Feature Request: Add health checks to proxy container

[arschles]

I am starting up the cloudsql proxy as a sidecar to my app, and I notice that my app fails non-deterministically. After some trial and error, it appears that the proxy may take some time to start listening on its port(s). I believe that a Kubernetes healthcheck would solve this issue. Does the proxy support healthchecks?

[AthenaShi]

Please check this for healthcheck using Kubernetes:

#119

[arschles]

Hi @AthenaShi - thanks for the reply and reference. Would you be open to a PR to add an HTTP endpoint that adds an endpoint that can be used as a health check?

[AthenaShi]

Ya, sure!

[ahmetb]

We might incorporate this in the official sample manifest.

In the meanwhile can you give this livenessProbe with exec a try: #119 (comment) and see if it fixes your intermittent issues?

[abstrctn]

Thanks @ahmetb, the livenessProbe you linked to is helpful for ensuring the proxy is listening on a given port.

However, I wonder if a "healthy" state should also involve testing the connection to the instance. A few cases where the netcat test would succeed, but an application could not actually connect:

• A typo in the -instances argument pointing at a nonexistant instance name
• The database instance going down for any reason
• The service account used by the proxy losing IAM permissions to access the instance

I'm not sure whether those are appropriate to test continually while running the proxy, but I know we'd find use from testing it within a readinessProbe, to ensure any edits to the configuration don't break connectivity. (This is especially useful if running the proxy as a service, when you don't have application code and user credentials to test the connection.)

[ahmetb]

livenessProbe you linked to is helpful for ensuring the proxy is listening

you can probably ship the mysql CLI with the image (or write a small script alongside your container) that makes a simple ping query to the database and exits. You can still use "exec" livenessProbe for this purpose and it's a somewhat common practice to test liveness of non-HTTP systems via Kubernetes.

[caquinovs]

@arschles out of curiosity, when you proxy may take some time to start listening on its ports, how much time have you observed so far? I'm starting to do some troubleshooting on my own pods, and I still have yet to confirm it, but cloudsql proxy logging indicates that it could be hours after container start.

[bcbrockway]

We were using #119 (comment) as our healthcheck but nc has been removed from the v1.16 release. What's the official recommendation now? Seems a bit crazy that we should have to create our own image to add livenessProbe support when using gce-proxy as a sidecar is a well-documented use-case.