Support taint tollerations

[locomoco28]

Is your feature request related to a problem? Please describe

Gitpod only gets deployed on nodes with specific labels, but other workloads can be deployed on these nodes aswell. To fix this, the nodes can have taints applied to them. The Gitpod config should imo have an easy option to set taint tolerations instead of relying on post-processing the rendered manifest, since that's not officially support and not recommended.

Describe the behaviour you'd like

A config in the gitpod.config.yaml that specifies the tolerations for the Gitpod workloads.

# Following the recommendation to group the meta and ide nodes
# and the workspace nodes separately
# We can set the value to "gitpod" on both as we can control which
# node gets meta/ide workloads, and which workspace workloads,
# using affinity labels, but it might still be useful for some people
# to have separate tolerations for both (like differentiating VM sizes)
taintTolerations:
  meta: # meta/ide
    - name: app
      value: gitpod
      effect: NoSchedule
  workspaces: # workspace
    - name: app
      value: gitpod
      operator: Equal
      effect: NoSchedule
    - name: size
      value: NDv2
      effect: NoSchedule

Example outputs

# apps/v1/Deployment minio
apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio
spec:
  template:
    spec:
      tolerations:
        - key: "app"
          value: "gitpod"
          effect: "NoSchedule"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: gitpod.io/workload_meta
                    operator: Exists
---
# apps/v1/Deployment ws-manager
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ws-manager
spec:
  template:
    spec:
      tolerations:
        - key: "app"
          operator: "Equal"
          value: "gitpod"
          effect: "NoSchedule"
        - key: "size"
          value: "NDv2"
          effect: "NoSchedule"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: gitpod.io/workload_workspace_services
                    operator: Exists

Describe alternatives you've considered

Additional context

Relevant Discord Question: https://discord.com/channels/816244985187008514/879915120510267412/969161894101389312

Ticket summary

MrSimonEmms — Today at 11:50
There are labels defined in the CLI - have you added them to the nodes? If you run gitpod-installer validate cluster --kubeconfig /path/to/kubeconfig --config /path/config, it'll tell you if you've done it right

locomoco — Today at 13:02
I did, but I also have other workloads being deployed on these nodes. I'd like to taint my Gitpod nodes so only Gitpod workloads get deployed on them

MrSimonEmms — Today at 13:04
We don't currently support taints, just the labels
https://discord.com/channels/816244985187008514/948592762834534490/950066965404328046

locomoco — Today at 13:04
So I have to manually go through the rendered YAML and set the taint tollerations manually every time I re-render the config?

MrSimonEmms — Today at 13:05
If that's what you want to do, yes. As it's currently not supported, you'll need to refer to the post-processing section in the "advanced" part of the readme

locomoco — Today at 13:06
Ooh that's great, I missed the post-processing part. That actually helps me a lot, thank you :pepelove~1:

MrSimonEmms — Today at 13:08
Remember - HERE BE DRAGONS!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vyas-n]

Commenting so that this doesn't get stale,

I run into exactly this issue when running in my k8s cluster. I run into resource allocation issues unless I either dedicate a k8s cluster just for GitPod (what I'm currently doing) or I start killing off the pods that are currently running on the GitPod nodes (not good and not scalable).

[adrienthebo]

Thanks for submitting this! We see this as an issue coming up and will look into it; we may adopt something like this to support our reference architectures.

[cedvan]

up ?