example rabbitmq-server.service.example systemd service unit should automatically restart rmq

[rgl]

The rabbitmq-server.service.example is not configured to automatically restart when there is an error. Is there a reason not to?

i.e. it should contain the line Restart=always or Restart=on-failure and maybe RestartSec=10.

For reference see the Restart= documentation.

[lukebakken]

Is there a reason not to?

I suspect it is because there are (rare) failure situations where auto-restart is not a good solution, or that an auto-restart may clobber data that could be used to diagnose the failure.

@michaelklishin probably has more historical information about this.

[michaelklishin]

@rgl there is no big idea behind not having that line. I recall a similar discussion and a similar question about the Windows service. Feel free to submit a PR that adds Restart=on-failure against the stable branch. I think we should also add a StartLimitIntervalSec value, say, 5 seconds?

[lukebakken]

We may want to use StartLimitIntervalSec= and StartLimitBurst= to prevent too many restarts.

There is some discussion here.

[michaelklishin]

@rgl would you like to submit a PR or should our team handle this?

[rgl]

Ah, those are good points! Now I'm in a catch-22 "to restart, or not to restart, that is the question" moment!

But, I'm lingering towards starting indefinitely... with Restart=always, StartLimitIntervalSec=0 (to disable rate limiting and keep restarting forever) and RestartSec=10 (restart after 10 seconds for not burning the cpu/logs in case it really keeps restarting forever). Something like:

cat >/etc/systemd/system/test.service <<'EOF'
[Unit]
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
ExecStart=/bin/bash -c "date '+%F %T.%%N';exit 1;"
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable test
systemctl start test
journalctl --follow -u test
systemctl disable test

But I'm not really sure what should be the default for rabbitmq :-/

[michaelklishin]

Restarting forever may be OK but not rate limiting (in particular, not limiting concurrency of possible restart attempts) sounds like a very bad idea to me. I 👎 setting StartLimitIntervalSec to 0. A single restart attempt every 10 seconds seems reasonable to me.

[rgl]

But is there other way to keep restarting forever without disabling rate limiting with StartLimitIntervalSec=0?

[michaelklishin]

Why do we need to restart forever? Is it really such a good idea? I don't know.

I'm no systemd expert but my reading of the docs is that they have the same "restart intensity" settings as in Erlang: a time interval and how many restart attempts ("bursts") in that time frame are considered to be reasonable.

The only caveat is

they apply to all kinds of starts (including manual)

and

Note that systemctl reset-failed will cause the restart rate counter for a service to be flushed,
which is useful if the administrator wants to manually start a unit and the start limit interferes with that

So it sounds like this will effectively restart forever unless things are so broken that it restarts more than N times in T seconds.

In which case maybe "1 restart a second" or two seconds should be the limit. Because…

Note that this rate-limiting is enforced after any unit condition checks are executed,
and hence unit activations with failing conditions are not counted by this rate limiting