Epic: [installer] Rollout migrations first

[geropl]

Problem

Currently, installer render renders all YAML files for all components. If they are applied in one step (as it's currently the case for A Self-Hosted installations as well as B current ArgoCD applications), migrations and code changes are applied in parallel. Depending on timings, this might cause code to hit an unprepared DB, which might end up in all kind of user-errors.

Situation

So far, this was not an issue because:

• A Self-Hosted installation history is not that long so migrations were near-instant
• B SaaS is using separate deploy jobs which handle migrations.

Motivation

The SaaS handling is suboptimal in terms of complexity introduced, and we would love to move to a pure ArgoCD-setup without multiple layers of external wrappers.

Also, there might be a chance to solve this problem for Self-Hosted (which will become more apparent as SH installations grow and age) with the same/similar approach.

Suggested solution(s)

1. Use "hook" concepts

Helm - which we already use internally in the installer - as well as ArgoCD have the concept of using "hooks", which are designed to handle exactly that case.

We could:

1. make installer render the migration job with these annotations:

annotations:
    argocd.argoproj.io/hook: PreSync
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-1"
    "helm.sh/hook-delete-policy": hook-succeeded

2. B ArgoCD: Make sure ArgoCD is configured to stop on a failed PreSync hook
3. A Self-Hosted: Could use helm to apply the new yaml (are we already doing this? 🤔 )

2. ???

Next steps

1. Clarify if Team Self-Hosted sees benefit here, and if it's worth tackling now
   1. also depends on: Are we already using helm to apply the new yaml? If yes, this might be a no-brainer
2. Check with Team Platform if they agree on the use of the PreSync hook

@mads-hartmann @corneliusludmann

[mads-hartmann]

@geropl This sounds great 🧡 Currently the Werft job also takes care of scaling down db-sync (ref) - would we need a separate hook for that?

[geropl]

Currently the Werft job also takes care of scaling down db-sync (ref) - would we need a separate hook for that?

Ah, you're right. I guess we do. 👍 Not sure if ArgoCD supports that, though. And maybe we could make db-sync smarter so we do not have to solve this on the infra level.

[mads-hartmann]

Not sure if ArgoCD supports that

At a quick glance there's nothing about ordering in the docs about hooks. We would have to look at bit closer to see what our options are if we go this route ☺️

And maybe we could make db-sync smarter so we do not have to solve this on the infra level.

I'd prefer this of course, but I don't have a sense of how much work that would be.

[meysholdt]

Hello!

Thanks for bringing up this issue. I think the question of how we migrate the database is critically important not only for SaaS-deployment experience, but also for the install/upgrade experience of Gitpod Self-Hosted. In the spirit of the delivery team, we should find a solution here that works for SaaS as well as for Self-Hosted.

There are plenty of blog posts around this topic and "Running database migrations when deploying to Kubernetes" introduces a concept which I think can work very well for us:

Proposed Solution:

• deploy a Kubernetes Job that runs the db-migrations
• all pods that need to access the database have an init container that waits until the db-migrations job has finished. Either by checking the job status or by polling the database version.

I think this leads to a nice Operations Experience because the operator persona can follow the db-migration status by following log output of the db-migration-job. It's also great that the operator persona can re-start the migration manually if there is the need to do so, e.g. after fixing a bogus database entry. This is an improvement compared to today, because running db-migrations no longer depends on restarting the deployment. Furthermore, it's a great operations experience that db-accessing pods don't become "ready" unless the db-migration has finished. It plays nicely with k8s rolling updates. I think this should also play nicely with long-running db-migrations because k8s rolling update keeps the old pods running until pods in the new version have become ready. This assumes all db-accessing deployments have a replica-count >1.

What remains is the question about db-sync. While it would be great to get rid of it, I don't know if that needs to be a prerequisite for this Epic. With the proposed migration strategy we need to consider that db-migraitons in US and EU cluster would run in any order, potentially at the same time or minutes apart. Proposed solution: db-sync should handle version-difference gracefully by pausing it's sync activity as long as the versions of the databases are not identical. If that's something db-sync can do, then the deploy job would no longer need to stop/start db-sync.