Epic: Allow for custom annotations & env variables on K8s Resources for Self-Hosted Installations

[lucasvaltl]

Summary

Allow users to customise to adapt their deployment of self-hosted to the specifics of the environment/cluster it is applied to.

Context

Customers are finding it difficult to sometimes use KOTS because it does not allow them to add custom annotations or env variables to K8s resources. The installer produces a fixed manifest and that then gets applied to the kubernetes cluster. Any changes made forces them to re-apply those changes after an update to Gitpod, which can lead to confusion.

Specific reasons they need to customise / make changes:

• Annotate service to use internal networking
• Add an environment variable to S3 when used as a registry storage backend (internal reference)

Value

Allow Kots to work in more setups, because it can be customised to environment specific circumstances.

Acceptance Criteria

• Customers can add generic annotations and env variables to resources via a configuration field in the KOTS UI.
• This is well documented in our documentation.

Measurement

• Even more customers start using kots

Proposed solution

Allow for generic annotations on resources - have a way to set this in the KOTS UI & Installer Config. Then apply together with the installation.

Alternatives considered

1. Open issues for the missing configuration options customers are doing currently with post-processing and solve this properly
2. Rely on replicated git-ops flow (requires a feature request from us) for post-processing.

Additional Context

• Likely related to Add Option to Set Proxy Service as ClusterIP Type in KOTS Admin UI #9981 and 0b599a8
• Examples of what customers are currently customising:

Tasks

• Apply dummy functions to all components #10853
• Allow for custom annotations #10826
• Allow for custom envvars #10902
• Allow for custom labels (stretch goal) #10903
• Allow customization of Helm components #10942
• Configure KOTS input #10910
• Validate the input object #10908
• Document for Installer #10840
• Document for website #10941

[corneliusludmann]

I doubt that we want to support post-processing in the near future. Instead, we should open issues for the missing configuration options customers are doing currently with post-processing and solve this properly. See also this internal comment.

[lucasvaltl]

Here is the current list of known example customisations that are being done by customers via post-processing,. which can partly now be done via experimental config OR which would benefit from having customisable annotations:

1. Change proxy service to a cluster IP instead of a loadbalancer type when using your own load balancer (this was changed via [installer] support service type ClusterIP for proxy #10537):

experimental:
  common:
    serviceConfig:
      proxy:
        serviceType: "ClusterIP" # case in-sensitive

2. Annotate proxy service with internal networking tag to force internal IP (was changed via 0b599a8)

experimental:
 webapp:
   proxy:
     serviceAnnotations:
        networking.gke.io/load-balancer-type: "Internal"

3. In S3, you need to add an environment variable for it to work as a registry because of old libs in distribution/registry.

echo "Gitpod: Add s3 endpoint"
    yq eval-all -i 'select(.kind == "Deployment" and .metadata.name == "registry").spec.template.spec.containers[].env += {"name": "REGISTRY_STORAGE_S3_REGIONENDPOINT", "value": "s3.eu-north-1.amazonaws.com"}' gitpod.yaml

4. Annotate ws-manager to allow for scraping not from promethues, but datadog (More info, internal)
5. if proxy has annotations for service.beta.kubernetes.io/aws-load-balancer-internal: 'true' as well as an annotation for the subnet the ELB should be in - it would save us from having to switch it to ClusterIp and for a customer to have to involve another team (source)
6. Set HTTP_PROXY env var (internal discussion)

[mrsimonemms]

This is superseded - ignore

I'm going to propose this as the new config structure:

experimental:
  annotations:
    common:
      key: value
    kind:
      <k8s_kind>:
        <component_name>:
          key1: value1

This would allow a common annotation on ALL components, but also fine-grained control of components based upon the kind (eg, deployment) and the component name (eg, ws-daemon). I've specifically added this as a separate config object (ie, not under experiment.webapp.kind.annotation) as this makes it a simpler, more isolated config.

I'm going to propose only applying this to deployments, daemonsets, statefulsets (and their pods) and also services. This will exclude the likes of jobs, networkpolicies, rolebindings and configmaps which I don't think will require much annotating.

Are we happy with this as a config definition and implementation? @lucasvaltl @corneliusludmann

[corneliusludmann]

I'm going to propose this as the new config structure

I think we should NOT put this under experimental since we want to support this officially. Or is there a good reason to have this first there and move it later?

Regarding the config object structure. Just a random idea. I would like a structure like this but it could be more difficult to parse. What do you think?

componentAnnotations:
  '*':
    '*':
      key1: value1   # all components
  services:
    '*':
      key2: value2   # all service components
  '*':
    server:
      key3: value3   # all components called 'server'
  services:
    'proxy':
      key4: value4   # only service called 'proxy'

Otherwise, I'm fine with your more simpler version of the structure. Maybe we can make annotations more specific and kind as opposed to common feels a little bit odd to me, however, I don't have a better idea.

[mrsimonemms]

Proposed Documentation

There are times when it is desirable for custom annotations, envvars and labels should be added to an installation.

Installer Config

This can be added to the root of the config.yaml. If nothing is passed in, no custom parameters are added in.

The structure is based upon the standard Kubernetes resource definition. For annotations and labels, these must match the apiVersion, the kind and the metadata.name - a wildcard * can be used to match all resources. For environment variables, the apiVersion and kind are ignored, as these are only implemented on containers. As before, the * wildcard can be used on metadata.name.

customization:
  - apiVersion: "*"
    kind: "*"
    metadata:
      name: "*"
      annotations:
        appliedToAll: value
        hello: world
      labels:
        appliedToAll: value
        hello: world
  - apiVersion: "apps/v1"
    kind: "Deployment"
    metadata:
      name: "ws-manager"
      annotations:
        hello: ws-manager
      labels:
        hello: ws-manager
    spec:
      env:
        - name: HELLO
          value: world

This example would generate the following spec (these are simplified for readability reasons):

---
# apps/v1/DaemonSet ws-daemon
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: gitpod # system-value
    component: ws-daemon # system-value
    appliedToAll: value
    hello: world
  annotations:
    appliedToAll: value
    hello: world
  name: ws-daemon
---
# apps/v1/Deployment ws-manager
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: gitpod
    component: ws-manager
    appliedToAll: value
    hello: ws-manager
  annotations:
    appliedToAll: value
    hello: ws-manager
  name: ws-manager
spec:
  template:
    spec:
      containers:
        - env:
          - name: HELLO
            value: world

In the event of multiple matches, the final matching customization would be applied. For that reason, it is a good idea to structure your customization from least to most specific. System-generated values will never be overridden.

Validation

• All annotations and labels must be key/value pairs.
• All environment variables must be an array of envvars
• There must be a valid apiVersion, kind and metadata.name

KOTS Config

This is a raw YAML object that will be applied to the Installer config as-is - it will be a file upload (similar to the patch) as it's easier to create YAML files in an IDE than it is in a plain <textarea>. The format is described as above.

We will attempt validation with a preflight check. It's not currently clear how this will work, but should be possible

[lucasvaltl]

Thanks for writing this up @mrsimonemms ! Helps me understand how this would work in context I hope you found it helpful as well!

This raises a couple questions (happy to jump on a call):

• Is the way to write the config K8s standard or is it something we came up with? Context: just trying to understand how familiar users would be with this. Although granted it does not look complicated.
• What would our error handling look like, for example if someone were to have the wrong formatting?
• How would one apply this in KOTS? From the comments above, there would be a separate field (in the Kots admin ui) called installer config?
• This requires intimate knowledge of the components that Gitpod uses right? I think this is fine though - if you are doing custom annotations, you will know what you want to annotate.