Configure behavior per node through annotations

[olemarkus]

In clusters with multiple ASGs user for different types of workloads, it would be nice to be able to configure more of the NTH behavior through annotations rather than globally for the nodes the given daemonset is running on.
Cluster Autoscaler already has something similar where one can add annotations on nodes and pods preventing evictions/downscale.

For our use case, annotations for whether to cordon or drain is enough (both on maintenance events and spots). But I can imagine some may want something like the safe-to-evict annotation on pod level too.

[jgournet]

Just found this ticket - and yes, we would really really appreciate the "safe-to-evict" label on pods.
Some of our pods cannot have 2 running instances at the same time (done via a centralized lock file).
currently, NTH holds up a node termination because of that one deployment: the new pod just cannot start while the old one is still running.