Epic: [Self-Hosted] Enable a way to persist Gitpod K8s Manifests for Disaster recovery Purposes

[lucasvaltl]

Summary

Find a way to persist the k8s ymls needed to install (kubectl apply) Gitpod in a secure location in order to be able to use these to quickly re-install Gitpod in its last known state in the event of a disaster.

Context

In the event of an unforeseen event where, e.g. your entire cluster running Gitpod goes down, you need to be able to quickly re-instate Gitpod in its last known state. The actual application state and the workspace state is usually saved externally, however one also needs to have the k8s manifests used to install Gitpod. Otherwise one would have to go through the entire installation flow manually, which would be prone to human error especially in a stressful situation.

Value

• Speed up your Recovery Time Objective (RTO) and the risk of human error while doing a recovery (you do not need ti manually insert values into an installation UI)

Acceptance Criteria

• The K8s ymls required to install (apply) gitpod can be backed up outside of the cluster
• Ideally this is version controlled, i.e. this lives in a git repo (Git Ops Workflow)
• Documentation

Measurement

• Adoption by customers.

Potential solutions

• Enable KOTS Gitops workflow (we have an FR open for this)
• Enable a GitOps workflow ourselves
• One way would be to explicitly document the configuration and the kots CLI (non ui) installation path and then expect users to use this.
• Download config map in a regular cadence
• Rely on Replicated's backup and restore, with the downside that it would make the user assume we are backing up all data (i.e. also data stored in external dependencies) when in fact we are only backing up the application k8s manifest data. We cannot change the UI here, so it will be hard to message this.

Issues

• Document how to configure KOTS for a GitOps workflow #11069

Additional Context:

• Related issue (private)
• Allow for Replicated Snapshot, Rollback, & Restore #9926
• Docs: How to rollback, Backups and Disaster Recovery (Operational Guides Section) #10515
• Docs: Gitpod and Business Continuity (Concepts & Background section) #10516
• Document how to configure KOTS for a GitOps workflow #11069

[corneliusludmann]

In my opinion, the best way would be to have proper GitOps support. However, supporting this properly seems not that easy.

---

The “manual” way would be the following, IMHO:

kubectl kots get config --namespace <namespace> --appslug gitpod > config.yaml

→ gives you the current app config from KOTS that you can save as a file.

kubectl kots get app gitpod --namespace <namespace> -o json

→ gives you the version label of the currently installed Gitpod version:

[
    {
        "slug": "gitpod",
        "state": "ready",
        "versionlabel": "2022.6.0"
    }
]

To restore your Gitpod installation on a new cluster, run:

kubectl kots install gitpod \
        --app-version-label=<versionlabel>
	--namespace <namespace> \
        --name gitpod \
        --shared-password <admin-console-password> \
	--license-file license.yaml \
        --no-port-forward \
        --config-values config.yaml

Is this, what you are looking for, @lucasvaltl?

[lucasvaltl]

Closing this as it is done via #11971