Allow for Replicated Snapshot, Rollback, & Restore

[lucasvaltl]

Is your feature request related to a problem? Please describe

As a customer, I want to be able to rollback after I upgrade my gitpod installation using replicated. I also want to be able to periodically backup my application to help with disaster recovery.

Describe the behaviour you'd like

• Fully implement Replicate's Snapshot, Rollback, & Restore feature to work with and for Gitpod

Describe alternatives you've considered

• none

Additional context

• More info on the feature in Replicate's blog post

Dependencies

• Change the Installer customisation to not update the selector labels #11952
• https://github.com/gitpod-io/website/issues/2565

[mrsimonemms]

@lucasvaltl what's the expectation about what we backup here?

My assumption is that the in-cluster dependencies (MySQL, Minio, Registry) will be backed up and any external dependencies are excluded. My reasoning is that these resources will often have a simpler backup method and any backup we wish to provide will be:

1. download all the data to the k8s cluster
2. upload the data to the backup resource

This seems to be putting an unnecessary strain on the cluster.

We also don't really have much in the way of persistent volumes. We have:

• MySQL in-cluster *
• Minio *
• Registry *
• KOTS resources
• Redis

The *ed resources will already be backed up, Redis doesn't require storage and it's arguable whether the KOTS resources need it

[lucasvaltl]

Thanks for raising this @mrsimonemms and being critical. I actually agree:

• For external dependencies, we should rely on and point to the best practice backup method said external dependency
• For in-cluster dependencies, we in general have less of a reliability expectation. These are to be used in PoCs etc, not really in a production setting.

Given that most if not all of the relevant data is stored in the above mentioned dependencies, what remains in terms of backup is the configuration of your Gitpod instance. This is actually something worth backing up (to minimise time to recovery), but for now it is not worth the cost of implementing Replicated's entire rollback/restore/snapshot feature. The added downside of doing the former is that this replicated feature assumes a full back up (and this is mentioned in the UX), and when we in reality only backup the configuration this can lead to misunderstandings with users with severe consequences.

We should still think about how we can help persist the installation configuration of Gitpod:

• One way would be to explicitly document the configuration and the kots CLI (non ui) installation path
• Maybe we have other ideas? Raised tracking issue for this: [Self-Hosted] Way to persist Gitpod Installation Config for Disaster Recovery Purposes #10549

Given the above, I will close this ticket. In spirit, it will be replaced by #10515. Creating this piece of documentation should still allow us to help our users with their backup strategy, while not creating the impression that we are backing up everything for them and rely on existing best practices for backups of the external dependencies.

[mrsimonemms]

Reopening issue after customer requests for this feature.

[mrsimonemms]

I've had a bit of a play with getting the in-cluster DB and storage backed up (see #12076 and #12045) but this is proving quite difficult due to using the Installer to do the deployment. Truthfully, the problem is a little nebulus and hard to explain, but I'll do my best - it's not impossible to solve, it's just that in our current implementation is would be a bit hacky.

1. To backup a resource, it must have certain labels applied to it - easy to do with our customisation in the Installer config
2. To backup the DB, it must have an annotation applied to it with the backup command - easy again
3. At the point of restoration, the DB pod fails to be restored because the service account doesn't exist - as the customisation doesn't extend to service accounts, this would require the use of yq to post-process the YAML
4. If we're post-processing the YAML, we may as well apply it to everything. Easy to do for the main resource, but certain things (ie, anything with a pod) needs it on the pod as well. This is where it starts getting a bit hacky

The problem is not that it's impossible (or even THAT hard), it's that it would introduce a lot of post-processing and customisation to the Installer job that would be hard to test and may introduce weird regressions. To my mind, the risk/cost of adding it far outweighs the benefits of having it - as this would only be available for in-cluster dependencies, any enterprise deployment will likely use cloud dependencies for their data persistence, which has more sophisticated native backup solutions.