CAPI quickstart doesn't work if existing cluster has taints

[andrewsykim]

What steps did you take and what happened:
I tried to run through the CAPI quick start for vSphere and noticed that the CAPI components and the infrastructure-components.yaml file from all providers don't tolerate any taints, so an existing cluster built with kubeadm doesn't work cause it applies the master taints for control plane nodes.

Tolerating all taints is probably not ok, but I think practically we should tolerate the well-known taints at least for the quick start which probably applies for the 80% of users that try it. Or a kubeadm-based cluster at the very least.

What did you expect to happen:
The quick start to work quickly.

Anything else you would like to add:

Environment:

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

/kind bug

[ncdc]

Are you saying your bootstrap/management cluster has the taints?

[andrewsykim]

yes, for CAPA and CAPV it's patched (via kustomize) as part of the examples directory. e.g. https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/master/examples/provider-components/manager_tolerations_patch.yaml.

But those tolerations aren't bundled into infrastructure-components.yaml as far as I can tell.

[ncdc]

I'm wondering about the taints in your cluster (not the tolerations)?

[andrewsykim]

Happy to work on this once we agree on a set of tolerations.

It has node-role.kubernetes.io/master - which is added by default by kubeadm.

[ncdc]

This is probably something we need to fix, and I'm trying to gather the facts. The quickstart works if you use a kind bootstrap cluster. I take it you didn't use kind?

[andrewsykim]

I didn't use Kind and I just verified that Kind does not apply any taints on nodes which explains the quick start working only for Kind.

[andrewsykim]

I think just tolerating node-role.kubernetes.io/master is a good start

[detiber]

If we want clusterctl to continue to work, then tolerating node-role.kubernetes.io/master would probably be a minimum requirement.

[andrewsykim]

If we want clusterctl to continue to work, then tolerating node-role.kubernetes.io/master would probably be a minimum requirement.

Just to be clear, I think most examples out there of clusterctl does include a toleration of node-role.kubernetes.io/master, but if you are bringing your own cluster it does not tolerate it even though most clusters may have the same taints as one that is bootstrapped by clusterctl.

[andrewsykim]

/assign

[vincepri]

The assumption of the quickstart is also that you wouldn’t run the controllers on a control plane node, given that they can run on a simple node.

Is this something that can maybe go in an advanced section?

[detiber]

The assumption of the quickstart is also that you wouldn’t run the controllers on a control plane node, given that they can run on a simple node.

Should that be the case, though? For example, if running on a Cluster API provisioned cluster in AWS, this would require an IAM profile for the Nodes that has a much higher set of permissions than we would generally require today.

[andrewsykim]

The assumption of the quickstart is also that you wouldn’t run the controllers on a control plane node, given that they can run on a simple node.

I think in reality most management clusters would only have control plane nodes though? Especially if we're advising users to not run workloads on a management cluster