Implement HEALTHCHECK instruction

[iBobik]

There is a new Dockerfile instruction HEALTHCHECK what is useful for detecting if container is still starting (and other should wait) or unhealthy.

It it enought to write something like this?

HEALTHCHECK CMD mysql --user=$MYSQL_USER --password=$MYSQL_PASSWORD -e "show databases;"

[iBobik]

Maybe to make it working also if MYSQL_USER or MYSQL_PASSWORD is not filled → use root. Then if MYSQL_ALLOW_EMPTY_PASSWORD and other related settings.

Any ideas?

[tianon]

docker-library/cassandra#76 (comment) has some good points regarding adding default health checks to official images 👍

(https://github.com/docker-library/healthcheck also has copies of several images with reasonably-generic HEALTHCHECK instructions added)

[yosifkit]

Current stance is still the same as linked above in cassandra.

I do not feel that generalized healthchecks on the official images are really that useful.

• users will have their own idea of what is "healthy"
• it does not actually test that the service is listening to connections outside of localhost (see https://github.com/docker-library/healthcheck for some examples that do more than what's proposed here, including attempting to check whether the service is listening remotely)
• some of the Official Images even purposely start in a localhost only mode for database initialization and then kill and start the main service with full network availability
• after upgrading their images, current users will have extra unexpected load on their systems for healthchecks they don't necessarily need/want and may be unaware of

Closing old issue.

[SalathielGenese]

This' how is made it

mariadb:
   image: mariadb
   restart: always
   volumes:
     - './mariadb/:/var/lib/mysql/'
   healthcheck:
     test: ["CMD", "mysqladmin", "ping", "--silent"]

[ebuildy]

Pay attention, if you use the /docker-entrypoint-initdb.d folder to run some SQL scripts at runtime, it may be useful to test if data is ready instead of just the server.

[katsar0v]

Pay attention, if you use the /docker-entrypoint-initdb.d folder to run some SQL scripts at runtime, it may be useful to test if data is ready instead of just the server.

That's an issue. Can we reopen this issue? If you have entrypoint sql or sh, the healthcheck from @SalathielGenese does not work

[grooverdan]

ideas welcome - https://jira.mariadb.org/browse/MDEV-25434

[Ajedi32]

FYI, it looks like there's now a healthcheck script built into the image, it's just not called in the Dockerfile for some reason. You can enable it like so:

services:
  mariadb:
     environment:
       MARIADB_MYSQL_LOCALHOST_USER: true
     healthcheck:
       test: ["CMD", "/usr/local/bin/healthcheck.sh", "--su-mysql", "--connect", "--innodb_initialized"]
  other_service:
    depends_on:
      mariadb:
        condition: service_healthy

For more details, and additional tests that can be enabled, see https://github.com/MariaDB/mariadb-docker/blob/master/healthcheck.sh

IMO this (or something similar) should be enabled by default. I'm a firm believer in setting reasonable defaults, and I don't think external services failing to start because docker-compose has no way of checking whether the service is healthy qualifies as "reasonable".

users will have their own idea of what is "healthy"

Users can customize the health check command if they want to. That's not a good reason for the default behavior to be "no health checks at all; just fail silently and don't give the container engine any indication that the service is unhealthy".

it does not actually test that the service is listening to connections outside of localhost

Okay. Again, that's not an argument for not having a basic healthcheck by default. If users want something more comprehensive than what Docker's internal health check mechanism can provide then they can always write it themselves.

some of the Official Images even purposely start in a localhost only mode for database initialization and then kill and start the main service with full network availability

All the more reason for those official images to have a proper health check predefined, so users don't try to write their own health check and discover that behavior the hard way. In images where it's applicable the default health check should detect "localhost only mode" and react appropriately. If programing that is too much effort for some reason, then leave out the health check on the images where that's a problem, and include a note about why. That'd be disappointing, but no worse than the current status quo.

after upgrading their images, current users will have extra unexpected load on their systems for healthchecks they don't necessarily need/want and may be unaware of

It's a healthcheck, not a stress test. A simple healthcheck script running twice a minute should not impose any significant additional load on a system. If for whatever reason the user wants something more complicated that is resource intensive, then let them enable that themselves so they're aware of the trade-off.

[Luc45]

This' how is made it

mariadb:
   image: mariadb
   restart: always
   volumes:
     - './mariadb/:/var/lib/mysql/'
   healthcheck:
     test: ["CMD", "mysqladmin", "ping", "--silent"]

This is wrong, please edit your reply to not induce others into error.

If you do this, it will do the ping with root user with an empty password, which can fail (silently) with this message:

root@b8ab90e765c2:/# mysqladmin ping       
mysqladmin: connect to server at 'localhost' failed
error: 'Access denied for user 'root'@'localhost' (using password: NO)'
root@b8ab90e765c2:/# echo $?
0

As you can see, the access was denied, but the exit status code was still 0!

You need to specify the root password to use that command. For instance, if your root password is 123:

healthcheck:
    test: ["CMD-SHELL", "mysqladmin ping -P 3306 -p123 | grep 'mysqld is alive' || exit 1"]
    interval: 2s
    retries: 15

[Ajedi32]

@Luc45 Ping just checks the connection, not whether your credentials are correct. It will exit with 1 if the server is not running, which might be good enough depending on your goal. If you need something more complicated than that, see the example in my previous comment.

[jerdoe]

FYI, it looks like there's now a healthcheck script built into the image, it's just not called in the Dockerfile for some reason. You can enable it like so:

services:
  mariadb:
     environment:
       MARIADB_MYSQL_LOCALHOST_USER: true
     healthcheck:
       test: ["CMD", "/usr/local/bin/healthcheck.sh", "--su-mysql", "--connect", "--innodb_initialized"]
  other_service:
    depends_on:
      mariadb:
        condition: service_healthy

For more details, and additional tests that can be enabled, see https://github.com/MariaDB/mariadb-docker/blob/master/healthcheck.sh

When I use these settings, the healthcheck test result for option --connect is correctly reported but I get a warning in my logs every time the healthcheck.sh script is run :

root@0fe582a68cc7:/# healthcheck.sh --su-mysql --connect
root@0fe582a68cc7:/# echo $?
0

$ sudo docker compose logs -f mariadb
proj_mariadb-1  | 2022-05-08 21:10:02 1453 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: NO)
proj_mariadb-1  | 2022-05-08 21:10:32 1461 [Warning] Access denied for user 'root'@'127.0.0.1' (using password: NO)
proj_mariadb-1  | 2022-05-08 21:11:03 1463 [Warning] Access denied for user 'mysql'@'127.0.0.1' (using password: NO)
proj_mariadb-1  | 2022-05-08 21:11:05 1443 [Warning] Access denied for user 'mysql'@'127.0.0.1' (using password: NO)

Samely, if I enter the following command, the same warning is thrown in my docker compose logs and in my console :

root@0fe582a68cc7:/# gosu mysql mysql -h localhost --protocol tcp -e 'select 1'
ERROR 1045 (28000): Access denied for user 'mysql'@'127.0.0.1' (using password: NO)

but this would not throw the warning:

root@0fe582a68cc7:/# gosu mysql mysql -h localhost -e 'select 1'
+---+
| 1 |
+---+
| 1 |
+---+

Is that something which is expected or is there something wrong with my setup ?

[grooverdan]

MARIADB_MYSQL_LOCALHOST_USER creates mysql@localhost but the --connect is using a tcp connection so mysql@127.0.0.1 is the expected user. Tested locally and mysql@::1 was used.

Its possible to hide these with log_warnings=1.

I maybe think I'll just expand the MARIADB_MYSQL_LOCALHOST_USER to create the mysql@127.0.0.1/ mysql@::1 user. I hope it's not too confusing to have MARIADB_MYSQL_LOCALHOST_GRANTS not give the same grants on those users.

Thanks for your comment @jerdoe. Nothing is really wrong with your setup. Creating new issues is ok too.