ws-manager CA used for generating mTLS secrets needs to be cycled by April, 1st

[geropl]

The certificate we use as CA for generating our cluster internal certificates will expire on 1st of April, 2022.

A derivative (in-cluster secrets are called ws-manager-tls and ws-manager-client-tls) is used most prominently by ws-manager and all it's clients for mTLS on the ws-manger interface. This is important, because that's what we configure in the DBWorkspaceCluster database!

Steps we need to do before 1st of April, ideally within next week to not interfere with the Offsite:

• webapp: verify that we do not rely on those for (non-local) connections to ws-manager
• platform: create a new certificate, and place that in Google Secret manager
  • ⚠️ we must not run a TF script on the app clusters after this!
• workspace: deploy a new set of WorkspaceClusters, and register those in the DB
• workspace: shift workspace traffic to the new clusters
• platform: run TF scripts to update the CA in app clusters
• webapp: re-deploy webapp with new CA (incl. workspace components in app cluster)

@kylos101 For the new workspace cluster creation
@meysholdt @mads-hartmann for being aware of this, and the platform parts
@jldec @JanKoehnlein For team WebApp

@wulfthimm Thx for making us aware again! 🙏

[geropl]

Besides doing the actual migration we should also think about how to better handle this in the future.

[fullmetalrooster]

The secret in GCP which contains the certificate can be found at https://console.cloud.google.com/security/secret-manager/secret/gitpod-ws-manager-ca/versions?project=gitpod-191109

[fullmetalrooster]

The GCP secret is read in the Terraform module gitpod which is part of the module gitpod-meta. The module gitpod-meta is used for the deployment of the application-clusters prod-meta-eu01 and prod-meta-us01. When we have updated the secret and disabled the previous secret version in GCP we can deploy them to the meta-clusters by running the Terraform scripts for both application-clusters.

[fullmetalrooster]

I have not tested the behaviour changing the root-certificate in a ca-issuer. We have to make sure that the certificates that are generated by the ca-issuer updated after the exchange.
The certificate resources for certmanager are:

• messagebus-certificates-secret-gitpod
• ws-daemon-tls
• ws-manager-client-tls
• ws-manager-tls

After we verified that the newly created certificates are updated we can restart the services.

For clarification:
The ca-issuer uses the root-certificate, the certificate-resources request new certificates from the ca-issuer and the actual certificates are stored as secrets.

References:

• ca-issuer: https://cert-manager.io/docs/configuration/ca/
• certificate: https://cert-manager.io/docs/usage/certificate/

[fullmetalrooster]

@geropl Do we have a script to generate the root-certificate?

[geropl]

I have not tested the behaviour changing the root-certificate in a ca-issuer

@wulfthimm Yes, this is true. But also only relevant for the WebApp deployment. 👍

Do we have a script to generate the root-certificate?

No. I think I used openssl for the original one.

[geropl]

Regarding

Besides doing the actual migration we should also think about how to better handle this in the future.

In the future, we no longer require the certificate be shared between app and ws clusters because no longer statically configured ws-manager-bridge using the app-cluster client certs (assuming those validate against the ws-cluster cert chain as we did in the past).
Instead, we're using gpctl to configure workspace clusters at runtime. That automatically fetches the client certs from the ws cluster and puts them into the DB.