kapp-controller won't delete an application without valid service account or kubeconfig #416
Description
What steps did you take:
Create an application that is meant to be deployed to a cluster. Make some mistake that prevents cluster deployment, like for example mistype the secret name. This simple app will do it:
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
name: simple-app
namespace: default
spec:
cluster:
namespace: carvel-apps
kubeconfigSecretRef:
name: make-sure-this-does-not-exist
key: value
fetch:
- git:
url: https://github.com/vmware-tanzu/carvel-simple-app-on-kubernetes
ref: origin/develop
subPath: config-step-2-template
template:
- ytt: {}
deploy:
- kapp: {}
Try to deploy that application with kubectl
kubectl apply -f simple-app.yml
The application will go into a reconciliation failure mode. Finally try to force delete that application:
kubectl delete app.kappctrl.k14s.io/simple-app --force
Your process will get stuck forever or until the application gets manually reconciled.
What happened:
Under certain circumstances, when incurring a failure, kapp-controller will ignore requests to delete the application, even if those requests are explicit about forcing the deletion of the application. kapp-controller will always expect the operator to fix the application first, and only then will proceed with the deletion. It's important to note that this behavior does not happen with all the errors. If, for example, remote deployment fails due to the destination namespace not existing in the target cluster, then kapp-controller will fail but it does not expect the user to fix the problem and delete commands will succeed.
What did you expect:
We would expect kapp-controller to delete the application when instructed to do it with the force flag. I think there might be an argument to keep the existing behavior of blocking CR deletion in certain circumstances to prevent possible inconsistencies. However, when the force flag is explicitly passed, I would expect the controller to go ahead and delete. It is also important to note that in this very simple example I did paste above, there haven't been any resources deployed anywhere. kapp-controller could be smart enough to detect this and proceed with the regular deletion.
Anything else you would like to add:
This blocking on delete behavior works consistently with the declarative mindset. Specially if we assume that there is the figure of an operator that might need to eventually jump in to manually resolve issues or applications that somehow are stuck. However, this does not fit very well the model of an automated deployment system that might deploy hundreds of carvel applications every day and that needs to be resilient to issues that can't be controlled, like for example an user mistyping some configuration, tokens not properly configured and expired, changes in the targets cluster, etc. Systems, by definition, will fail and what would happen today with the current design is that resources would start leaking until someone gets alerted and can manually get involved. And at that moment there could be dozens or hundreds of apps that would need to be manually reconciled.
I believe by adding a way to force deletion alleviates the concerns about the use case above.
Vote on this request
This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.
👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"
We are also happy to receive and review Pull Requests if you want to help working on this issue.