Add-ons management Race condition with Node Groups

[jmickle]

Description

During module creation with Add-on "coredns" and a managed node group, there is the inevitable possibility of a race condition that I have hit a few times. Based on the random order of returned resources of spinning up the new the coredns add-on can potentially be attempted to be created before a manage node group which inherently creates a "wait for creation condition" which will fail out after 20 min. On a subsequent plan/apply given the ordering of resource creation has shifted, the node group will be created and then finally core-dns will destroy and recreate successfully.

Versions

• Terraform: latest

• Provider(s): latest

• Module: latest

Reproduction

Steps to reproduce the behavior:

N/A N/A N/A

Code Snippet to Reproduce

Expected behavior

CoreDNS addon should probably have a depends-on block for the node profile.

Actual behavior

Terminal Output Screenshot(s)

Additional context

Happy to submit a PR, I just have not had an opportunity to get familiar with the actual module code at the moment. I will happily attempt to get familiar with the module to PR it this weekend. However, wanted to open an issue so if someone more familiar with it knows it could easily be a 5 minute fix.

[bryantbiggs]

We cannot control the behavior of CoreDNS in this module - its simply a configuration within the addon resource.

[jmickle]

thinking out loud here, from a high glance perspective can't a depends block be added to the addon's module that depends on the node_group module be ran first? Of course that would have to make the assumption that a node group is being constructed as part of the configuration............

[bryantbiggs]

I believe this is only an issue when deploying EVERYTHING at the same time from zero. not great if you are trying to do ephemeral clusters and launching them from scratch all the time, but that seems like a niche use case

[wojtekszpunar]

There is an additional problem with it.
Before add-ons were moved into the module, I had a separate module which created those add-ons and everything worked quite quick. I mean, it took maybe 2 min or so.
Once we have moved to module managed CoreDNS add-on our cluster boot time takes extra 15 min or more due to I believe this race condition (I can see that deployment is stuck on add-on installation and in console add-on is in degraded state).
I believe there is a connection between nodes created and add-on going into degraded state when we install add-ons within module which do not wait for nodes (aws/containers-roadmap#1389).

This happens nearly each time when you deploy a fresh cluster.

[daroga0002]

maybe explicit dependency to addons relying on node groups will help here?

[jmickle]

Have to admit, I am stuck on a good approach to actually solve this. Solving this might require the module to make assumptions about the cluster that this module probably should not make since it relates to the idea of "enforcing" a way for the node group to be managed which might not necessarily be the desired approach.... Maybe it's best to strike out the addon's management from this module and make that an entirely different module?!

[jjhidalgar]

I have the addons separate from the module, so when I want to create a cluster I first run terraform apply with target=module.eks and then just terraform apply for some additional things including the addons, it works decent.

[bryantbiggs]

The examples all work, and yes there is an added wait period for CoreDNS - what is the exact issue we are trying to solve for here?

[wojtekszpunar]

I think the main objective here is to reduce time needed to install CoreDNS add-on. The problem I believe is with timing and when you start deploying a new cluster with this add-on enabled you can observe that it's going into degraded mode on AWS EKS Console.

I think the issue is with execution time. I mean add-on resource waits for EKS cluster (control plane) and then starts deploying CoreDNS. Problem here (I think) is that we do not have yet any nodes (nodegroups or any other) and internal kube-dns deployment is in pending state when we try to update it(relates to bug reported in AWS container roadmap).
This causes some problem on EKS add-on implementation and add-on goes into degraded mode and retrying to install, resulting in average 14-minute delay to install just that add-on.

Moving it to module might help but might not because depending on your choice of nodes you might have to have depend_on certain type of node types.
I had previously an implementation (before ver 18) where add-ons were part of an external module installing other Kubernetes resources and I have never seen this issue. Most likely because EKS created both Control Plane and Data Plane (nodes) and only then proceed with installation of add-ons and other resources.

I think the way to fix it is to temporarily remove this implementation or put some information about consequences and wait until AWS fixes the issue.
Another option is to insert depend_on into add-on resource to wait on data plane (which ever option you choose) which might be tricky.

[bryantbiggs]

right, so lets break this down to specifics though:

• this only happens when spinning up a brand new cluster and you are provisioning the CoreDNS at the same time as cluster creation
• the perceived issue from the statement above is that it "takes too long"

I don't know what the long term effects are if we add a depends_on, but it seems less than ideal because then we are saying the addons are mostly node addons (what if we have an addon that is geared towards the control plane and now it has to wait for nodes which could send things into a deadlock waiting)

I think there are two things here that work today without modification to the module:

1. Users can opt out of the module provided addon functionality and provide their own addon resource
2. Wait for some upstream resolution as identified in the link provided above [EKS Add-On] [CoreDNS]: Patched Add-On never recovers from 'Degraded' State aws/containers-roadmap#1389

[jmickle]

@bryantbiggs Apologies I have been OOO but, the perceived issue is not that it "takes too long" It's that you could potentially never be able to successfully run the create of a new EKS cluster using the addons block as a result of the necessity of having a managed node group to create the addon for coredns. If the addon attempts to create before the node group, the addon will always fail as it will never reach a "stable" state.

[jmickle]

Comments about the time of duration taking too long are scope creep on this issue report, the real issue is the dependency to create an addon for something like coredns that requires at minimum the default node group for kube-system. If the default node group is not created, OR, the addon attempts to be created in terraform BEFORE the node-group, terraform will inherently wait until the timeout to fail, and the EKS module will result in a "failed to create cluster state" resulting in the next run being a full new destroy and create. At which point you have to hope the ordering of how the resources are created happen in a correct order so that it will successfully create.

[jmickle]

Adding in here for further review: The issue is also present with the EKS_EBS_CSI_DRIVER.

I am really starting to think, this module should not do addons should be independent of this module.... As AWS continues to releases addons primarily ones from kubernetes-sig this will inherently become more of a problem.......

I also do not believe that we should be accepting the idea of run targets first and then run full apply.... that feels like defeat in the notion of IaC