[EKS] Support secondary CIDR

[MarcusNoble]

/kind feature

Describe the solution you'd like
The ability to create workload clusters on EKS that make use of secondary CIDR ranges (AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG in amazon-vpc-cni-k8s) to avoid the need of large IP allocation while still having worker nodes that a routeable over DirectConnect.

Ideally this should also be supported when using existing AWS infrastructure where the control plane and worker nodes make use of the provided VPC and subnets and cluster-api create the needed additional subnets and route tables for the secondary CIDR.

For this to work fully without intervention cluster-api would be required to reconfigure the aws-node daemonset that comes with new EKS clusters as well as adding some ENIConfig custom resources. This needs to be done prior to the worker nodes joining the cluster for the config to take effect.

Anything else you would like to add:
https://www.eksworkshop.com/beginner/160_advanced-networking/secondary_cidr/
https://aws.amazon.com/about-aws/whats-new/2018/10/amazon-eks-now-supports-additional-vpc-cidr-blocks/

[richardcase]

/area provider/eks

[MarcusNoble]

/assign
/lifecycle active

[randomvariable]

Would be good to do in a way that AWSCluster can also set up secondary CIDRs so that the controllers are somewhat in sync.

[MarcusNoble]

Yeah, that's what I'm thinking. Not 100% sure how/if non-EKS clusters would manage amazon-vpc-cni-k8s.

[randomvariable]

They may. @Sn0rt previously tried to get an example manifest to use the VPC CNI.

[Sn0rt]

you can try to install multi ENIConfig with your cluster, and set DS with env value as follow

        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
          value: "true"
        - name: AWS_VPC_K8S_CNI_EXTERNALSNAT
          value: "false"
        - name: AWS_VPC_K8S_CNI_EXCLUDE_SNAT_CIDRS
          value: 10.0.0.0/8
        - name: ENI_CONFIG_LABEL_DEF
          value: k8s.amazonaws.com/pod-ip-pool
        - name: AWS_VPC_K8S_CNI_LOGLEVEL
          value: DEBUG
        - name: AWS_VPC_K8S_CNI_VETHPREFIX
          value: eni
        - name: AWS_VPC_ENI_MTU
          value: "9001"
        - name: WARM_IP_TARGET
          value: "1"

set eniconfig as follow

$ kubectl  get eniconfig -n kube-system
NAME             AGE
10-71-122-0-24   34d
default          104d

as before setting, you can run the pod as per node per subnet

[MarcusNoble]

So, non-EKS clusters don't come with any CNI or similar (by design) so not really sure how much this should do for those cases.
My current thinking is to focus this on EKS only for now and cover non-EKS use cases in another issue. @randomvariable any objections to that?