UMBRELLA: design and refactor graceful termination

[tapih]

To shutdown controller gracefully, containers should be terminated after endpoints deletion.

If time.Sleep(someSeconds) is added between these lines, applications made with controller runtime can wait for endpoint deletion as default.
What do you think?

[alexeldeib]

See some of the previous discussions about graceful termination:

• Stopping individual controllers in a manager #730
• Support for stoppable managers #695
• Manager does not wait for Runnables to stop #350

I think a better approach than sleeping (and one that's been discussed a few times) is properly wiring up all the contexts (well, currently stop channels) and using either a wait group (for manager shutdown) or some sort of map + lock + shutdown mechanism (for manager shutdown + dynamically (un)loading controllers).

There's a tentative desire to replace all the stop channels with context plumbed all the way through -- graceful manager termination could plausibly be done in tandem with that change (or not, but seems natural).

[tapih]

Sorry for my poor explanation. What I meant to propose is that, by inserting sleep just after the signal handler, we can omit /bin/sleep in preStop hook of Kubernetes Pods.

When a Pod is shutting down, pod termination and endpoint deletion are executed at the same time.
The problem is that some requests cannot be accepted if the endpoint deletion does not finish while the pod termination finishes.
I meant by the word "graceful shutdown" in the previous comment to avoid this situation by ensuring that the pod is terminated after the endpoint is deleted.

One way to wait for endpoint deletion is to add the sleep command into preStop, but if a base image does not have the sleep command, this approach cannot be taken.
So, I would like to add time.Sleep(someSeconds) to resolve this problem at the framework level.

[vincepri]

/kind design

We'd need a design document in form of PR to the controller-runtime repository.

/help
/priority important-soon

[k8s-ci-robot]

@vincepri:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/kind design

We'd need a design document in form of PR to the controller-runtime repository.

/help
/priority important-soon

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.