[reference architecture] Investigate using tigera-operator instead of static calico manifests

[adrienthebo]

Is your feature request related to a problem? Please describe

In February the Calico EKS documentation changed the recommended installation path to use the tigera-operator. Users running through the EKS reference architecture note that this pattern no longer follows the Tigera or AWS recommendations and may follow setup instructions that leave Calico in a state that doesn't satisfy Gitpod's requirements.

Describe the behaviour you'd like

We need to answer the following questions and provide documentation backing up our decision to use either tigera-operator or calico-vxlan.yaml.

• What does the new tigera-operator do?
• How does it differ from running kubectl apply -f https://projectcalico.docs.tigera.io/manifests/calico-vxlan.yaml?
• Is tigera-operator supported?

[adrienthebo]

Tacking a comment onto this issue, we should also document why we're using Calico.

[adrienthebo]

Copying a comment from @mrzarquon:

one implication is the operator has an eviction budget of 0 so you have to force delete the EKS cluster using eksctl (don't know how this is in terraform), otherwise eksctl / cloudformations dies waiting for the nodegroups to empty before they are terminated

[Pothulapati]

There is also some documentation on how the operator is better 🤔

[Pothulapati]

From the docs, I can see the following:

What does the new tigera-operator do?

Calico is installed by an operator which manages the installation, upgrade, and general lifecycle of a Calico cluster. The operator is installed directly on the cluster as a Deployment, and is configured through one or more custom Kubernetes API resources.

Considering that both manifests & operator are supported (but operator is recommended), I would err on the side of starting wit the manifests and then figuring out a path to the operator if needed. This also matches with the current goal of getting upto speed with that existing guides. WDYT? Happy to use the operator if there are stronger reasons though.

[adrienthebo]

In the long term, having the tigera-operator managing CNI upgrades would be great and there may be other benefits that we've yet to discover. This issue is targeted at the reference architecture GA release so we've got a ways to go before we need to resolve this issue; we can shave other yaks first 😄

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.