Add metrics to show the number of nodes being tracked

[stevehipwell]

Describe the feature
I'd like to know the number of nodes that a NTH instance is currently managing and have it exported as a metric.

Is the feature request related to a problem?
After the tagging changes we had a defect where the tags weren't being set on the instances causing our nodes to no longer be managed by NTH; this metric would have meant it was easy to see that it wasn't configured correctly.

Describe alternatives you've considered
n/a

[phuhung273]

Hi @stevehipwell, do you mean that we only need this feature for SQS mode ?

[stevehipwell]

@phuhung273 yes this only makes sense for SQS mode.

[phuhung273]

My 1st idea is counting instances based on tag. But there is only 1 mandatory tag key=aws-node-termination-handler/managed which cannot cover:

• Case 1: account with multiple clusters
• Case 2: cluster with both standalone instance and ASG

Came up with 2nd idea:

• Step 1: kubectl get node to get all nodes belongs to that cluster (can cover 2 cases above)
• Step 2: Check how many of them have key=aws-node-termination-handler/managed

May I know your thought @stevehipwell

[stevehipwell]

@phuhung273 the managed tag passed into NTH should be exclusive to the K8s cluster as NTH will attempt to manage all EC2 instances with that tag. How about the following metrics which would allow us to detect misconfigurations.

• nth_managed_nodes - number of K8s nodes in the cluster with the NTH managed tag set
• nth_managed_instances - number of running EC2 instances with the NTH managed tag

[phuhung273]

@phuhung273 the managed tag passed into NTH should be exclusive to the K8s cluster as NTH will attempt to manage all EC2 instances with that tag. How about the following metrics which would allow us to detect misconfigurations.

• nth_managed_nodes - number of K8s nodes in the cluster with the NTH managed tag set
• nth_managed_instances - number of running EC2 instances with the NTH managed tag

Amazing @stevehipwell. Is this pseudo code good enough ?

for {
  nth_managed_nodes = (kubectl get node --filter tag=NTH)
  nth_managed_instances = (aws ec2 describe-instances --filter tag=NTH)
  sleep(5s)
}