ASG LifeCycle hook preforming non-stop drain

[doryer]

Describe the bug
After upgrading to v1.23.0 and enabling the ENABLE_ASG_LIFECYCLE_DRAINING Flag, it seems that the Pod running on the node ( IMDS Mode ) receiving non-stop events of the ASG LifeCycle hook and trying non-stop to drain the nodes ( 151 Adding new event to the event store logs on the same host ).
It also causing the the pod to log the all workers busy, waiting due to high amount of events in the queue.
In addition, there is typo in the log of the event adding to the queue description, it is written AST target lifecycle instead of ASG target lifecycle

Steps to reproduce
Enable ENABLE_ASG_LIFECYCLE_DRAINING flag and enable lifetime on the ASG.

Expected outcome
Get 1 event, drain the node only once and finish.

Application Logs
151 Log entries of Adding new event to the event store
Entries of all workers busy, waiting

Environment
Kubernetes 1.28

• NTH App Version: v1.23.0
• NTH Mode (IMDS/Queue processor): IMDS
• OS/Arch: Ubuntu 22.04
• Kubernetes version: v1.28
• Installation method: Chart

[Lu-David]

Thanks for raising this issue @doryer. Could you share the full helm install command you used? And also how you created your lifecycle hook? I believe ASG lifecycle hook is only supported by sqs mode but asg lifecycle state is supported in imds mode. Thank you!

[doryer]

---

Thanks for raising this issue @doryer. Could you share the full helm install command you used? And also how you created your lifecycle hook? I believe ASG lifecycle hook is only supported by sqs mode but asg lifecycle state is supported in imds mode. Thank you!

Regarding the LifeCycle hook, it is deployed with CONTINUE policy with heartbeat timeout of 300 ( without sending to SQS, we're using the IMDS mode ).

Regarding Helm, I'm not able to provide the full helm install command as we're using something else in-house, the flags we're using to configure imds mode:
completeLifecycleActionDelaySeconds: 90
enableSqsTerminationDraining: false
enableASGLifecycleDraining: true
but I would love to provide you any flags requested in order to debug the issue.

[Lu-David]

TLDR: ASG lifecycle hooks are not supported in IMDS mode

Thanks for sharing those details. I realize now that our README is slightly incorrect, so will update but currently IMDS only supports ASG Lifecycle states but not ASG lifecycle hooks

Currently, NTH will poll IMDS to get the target-lifecycle-state Reference.

NTH will detect node will be entering terminated state. So no matter what lifecycle hooks you have, NTH will ignore it in IMDS mode and try draining anyway.

My recommendation is probably stick with SQS mode for lifecycle hooks.
If you're able to share what your use case is, then perhaps we can find an alternative solution!

Sorry for the confusion.

[doryer]

TLDR: ASG lifecycle hooks are not supported in IMDS mode

Thanks for sharing those details. I realize now that our README is slightly incorrect, so will update but currently IMDS only supports ASG Lifecycle states but not ASG lifecycle hooks

Currently, NTH will poll IMDS to get the target-lifecycle-state Reference.

NTH will detect node will be entering terminated state. So no matter what lifecycle hooks you have, NTH will ignore it in IMDS mode and try draining anyway.

My recommendation is probably stick with SQS mode for lifecycle hooks. If you're able to share what your use case is, then perhaps we can find an alternative solution!

Sorry for the confusion.

So actually we've changed from SQS Mode to IMDS mode lately due to lack of performance in SQS mode causing us to suffer from a lot of un-gracful shutdowns ( maybe it can be opened on separate issue, but IMDS works perfect for us ).
Thanks for the clarification, you think it is something that can be implemented in the future same as Queue processor mode ?
Thank you !

[Lu-David]

Probably not in IMDS mode (seems like they would only work with SQS or SNS topic based on the put-lifecycle-hook). Could you create a new issue regarding ungraceful shutdowns in SQS mode? If there's a performance issue, we'd like to fix it!

[doryer]

@Lu-David I want also to share production incident we had as part of this flag which enabled by default
We've deployed to production by mistake without manually changing the flag to false in the chart ( as from this use case described in the thread, we don't want to use it as it not fits our use-case ).
In approx 1 hour we got around 10K events in each cluster with the description of AST target lifecycle state received. Instance will be terminated
It caused extremely high load on our k8s api-servers, caused them to crash in our production environments, Luckily some of them we're still functional and we was able to delete the daemonset in order to get back to normal.
After reverting to 1.22.1 everything went back fine. Also tried to reproduce it again in our staging environment with this flag enabled or disabled with the new 1.23.0 image and i was able to do that
Steps to re-produce:

1. Enable life-cycle hooks inside AWS ASGs
2. Deployed 1.23.0 version with ENABLE_ASG_LIFECYCLE_DRAINING set to true
3. Monitor K8S Control planes CPU & Memory, especially in high scale cluster

I suggest to turn this flag to default false and also improve documentation around it as it can cause for production incidents.

[LikithaVemulapalli]

We have released new version that will handle this issue. Appreciate for finding the issue
https://github.com/aws/aws-node-termination-handler/releases/tag/v1.23.1