Auto Scaling Group Suspend Processes functionality

[timothymahoney]

/kind feature

Describe the solution you'd like
Auto Scaling groups give operators the ability to suspend certain functions on a case-by-case basis

AWS doc here:
https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_SuspendProcesses.html

Go SDK doc here:
https://docs.aws.amazon.com/sdk-for-go/api/service/autoscaling/#AutoScaling.SuspendProcesses

It would be good to be able to perform actions against the default processes that run, but in particular it's valuable to be able to disable the AZRebalance process for Kubernetes clusters that run the cluster-autoscaler. They can thrash against each other spinning up and down nodes which causes significant instability that can be difficult to diagnose.

Anything else you would like to add:
I would expect to be able to set an array of the processes I want to disable, example:

---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSMachinePool
metadata:
  name: something
spec:
  suspendProcesses:
  - AZRebalance
  - etc

Thank you for all the excellent work done so far!

Environment:

• Cluster-api-provider-aws version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[richardcase]

/triage accepted
/priority backlog
/help

[k8s-ci-robot]

@richardcase:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/triage accepted
/priority backlog
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Skarlso]

/assign

[Skarlso]

There are two choices for UX here:

1. Add another field called resumedProccess that the user would like to have resumed because they were suspended before.
   • This approach has the benefit that it's explicit in nature and we don't have to rely on the retrieved settings.
2. We can get the currently suspended processes and if they don't match the existing setting we enable anything that isn't in the list of user-defined suspended services. We only do this if the update is required ie, there is a change in existing ASGs and currently suspended services.
   • This approach has the benefit that it's a nice UX since users don't have to keep track of what they suspend or resume. They just update the list as desired.

[Skarlso]

Further, since this requires an update to the API and we are about to cut a new version v1beta2 I suggest we hold off on this before the new version is available.

[Skarlso]

beta2 has been released, so I'm going to pick this up. Also, I'm going to go with the transparency of the AWS settings. Meaning I'll introduce two new fields. Suspend and Resume to make it explicit which ones have to be dealt with.

This has the benefit that it matches the AWS API call, however it's also a bit weird because we reconcile constantly. Meaning it will constantly try to suspend a process. And you will always have a suspend list in your config which might be weird.