[Feature & Proposal] Maintenance Mode

[bagasme]

We can schedule a maintenance window in case we perform maintenance tasks that requires taking offline Gitea instance (such as upgrading Gitea, migrating data, scaling hardware resources, and troubleshooting system issues).

Feature:

• Maintenance mode can be scheduled at least certain times ahead (30 minutes, 2 hours, etc.), or absolute date & time (e.g. at June 1 23:00). The window can also be scheduled now.
• When an instance is in maintenance mode, all HTTP and Git requests should return 503 Service Unavailable error code.
• Upon maintenance, all users would see a banner notifying about upcoming scheduled maintenance.
• Visiting the instance's site should return maintenance landing page.

[lunny]

will replace #9577

[sapk]

The subfunction for message/banner is already discussed under #2345

[bagasme]

@sapk so #2345 is dependency of this issue?

[justusbunsi]

Anyone working on this or #2345? I would like to learn Go a bit more and could try to implement this feature. It just might take a while. 😉

[sapk]

I mentioned the issue #2345 to link them and if possible already take in account the broader usage of message broadcast but this can be done in second time if not possible in one time.

@justusbunsi no one opened a PR linked to those two issues so don't hesitate to start. If you have question feel free to come ask them on discord.
Since you will need to change things in frontend and backend this guide can be helpful to start: https://docs.gitea.io/en-us/hacking-on-gitea/

[justusbunsi]

After some time diving into the code and inspect the admin area, I would add a maintenance mode section inside the admin dashboard below the actions section. Is this a good place to have this?

Regardless of the configuration location: As proposed above it would allow manually enabling/disabling maintenance immediately, defining a message that will be displayed at the top of each page and specifying a datetime range to schedule the maintenance. Seconds will be trimmed.

What I am currently struggling with is where to store these data.

[lunny]

I think you could create a new table to store the message.

[wxiaoguang]

I think it's not feasible at the moment.
(update: it could be done by using designed rules to block all writing requests, only leave a few requests open for admin login/management, but the effort paid for it seems not trivial, while the benefit is trivial)

Gitea is a single-binary app, and no cluster support at the moment. When you are upgrading Gitea, migrating data, scaling hardware resources, and troubleshooting system issues, Gitea itself usually is not running.

You could setup a nginx (or some other proxy) to do the 503 response or show a banner.

[lunny]

feasible

But you can run serval instances and keep at least one running.

[wxiaoguang]

feasible

But you can run serval instances and keep at least one running.

Then, what's the benefit? The only running instance should stop all cron tasks, clear queue, stop all sessions and just tell user a 503 response code? And if you are running several Gitea instances, isn't there a nginx-like proxy before them? then why not just let the nginx return 503 .....

But the effort paid for it seems not trivial, while the benefit is trivial. And the situation might be worse for Windows SQLite users, as long as the database is in use, it could not be backuped.

[McNetic]

I also think the feature proposal is sensible.

• Firstly, it's cumbersome to have to manipulate a reverse proxy when it's normally only configured to do a simple forward.
• Secondly, the maintenance mode helps in shutting down the service without interupting running tasks and also provided a possibility to announce the downtime before it actually happens
• Thirdly, (and to me most importantly) it would also help for backup purposes even in a single instance installation, as it would put the repositories and the database in a state where a clean backup could be made, without shutting the service down completety and with a possibility to inform the users whats going on.

[m-ueberall]

• Secondly, the maintenance mode helps in shutting down the service without interupting running tasks and also provided a possibility to announce the downtime before it actually happens

To extend on the above: A maintenance mode does not (should not) necessarily require shutting down the service (read: terminate all processes, including the main one). And that's exactly what makes a huge difference w.r.t. monitoring/watchdog solutions that are "usually eager to restart one or more instances (in a cluster) if Gitea isn't up and running".
If a maintenance mode basically signals "I'm alive, restriction of certain functionality atm is intentional" instead – regardless of what will be restricted –, this does not require additional, solution-specific checks/rules on the outside.

Also, with respect to the third item above (backup purposes), switching one or more (connected) Gitea instances into maintenance mode in a defined manner (e.g. by flushing certain caches and/or waiting for some operations to finish in case of a specified "grace period") allows for external scripts/events to make use of a defined state for a series of actions that aren't/cannot be considered atomic together.
One example would be a backup consisting of dumping the database and creating a snapshot of the file system using two different tools (say, zfs [snapshot …] followed by mariabackup [--backup …]) basically at any given point in time after/before toggling maintenance mode–usually for a mere couple of seconds as "post-processing" said backup/copy does not require any restrictions anymore.