Allow for Replicated Snapshot, Rollback, & Restore

[lucasvaltl]

Is your feature request related to a problem? Please describe

As a customer, I want to be able to rollback after I upgrade my gitpod installation using replicated. I also want to be able to periodically backup my application to help with disaster recovery.

Describe the behaviour you'd like

• Fully implement Replicate's Snapshot, Rollback, & Restore feature to work with and for Gitpod

Describe alternatives you've considered

• none

Additional context

• More info on the feature in Replicate's blog post

Dependencies

• Change the Installer customisation to not update the selector labels #11952
• https://github.com/gitpod-io/website/issues/2565

[mrsimonemms]

@lucasvaltl what's the expectation about what we backup here?

My assumption is that the in-cluster dependencies (MySQL, Minio, Registry) will be backed up and any external dependencies are excluded. My reasoning is that these resources will often have a simpler backup method and any backup we wish to provide will be:

1. download all the data to the k8s cluster
2. upload the data to the backup resource

This seems to be putting an unnecessary strain on the cluster.

We also don't really have much in the way of persistent volumes. We have:

• MySQL in-cluster *
• Minio *
• Registry *
• KOTS resources
• Redis

The *ed resources will already be backed up, Redis doesn't require storage and it's arguable whether the KOTS resources need it

[lucasvaltl]

Thanks for raising this @mrsimonemms and being critical. I actually agree:

• For external dependencies, we should rely on and point to the best practice backup method said external dependency
• For in-cluster dependencies, we in general have less of a reliability expectation. These are to be used in PoCs etc, not really in a production setting.

Given that most if not all of the relevant data is stored in the above mentioned dependencies, what remains in terms of backup is the configuration of your Gitpod instance. This is actually something worth backing up (to minimise time to recovery), but for now it is not worth the cost of implementing Replicated's entire rollback/restore/snapshot feature. The added downside of doing the former is that this replicated feature assumes a full back up (and this is mentioned in the UX), and when we in reality only backup the configuration this can lead to misunderstandings with users with severe consequences.

We should still think about how we can help persist the installation configuration of Gitpod:

• One way would be to explicitly document the configuration and the kots CLI (non ui) installation path
• Maybe we have other ideas? Raised tracking issue for this: [Self-Hosted] Way to persist Gitpod Installation Config for Disaster Recovery Purposes #10549

Given the above, I will close this ticket. In spirit, it will be replaced by #10515. Creating this piece of documentation should still allow us to help our users with their backup strategy, while not creating the impression that we are backing up everything for them and rely on existing best practices for backups of the external dependencies.