After the AKS node image was upgraded, the csi driver daemonset csi-blob-node failed to start

[LeoandHenry]

1. Background:
   The customer used helm to install the csi driver before, and found that the pod of CSI-BLOB-Node could not start normally after an AKS node image upgrade.
   Installation link for customer reference: https://github.com/kubernetes-sigs/blob-csi-driver/tree/master/charts

2. AKS Cluster Information
   1)cluster version:1.27.9
   2)CSI driver version:1.23.3
   3)AKS node image version:AKSUbuntu-2204gen2containerd-202407.03.0
   4）Storage class：
   
   5）daemonset: csi-blob-node
   

3.Problem screenshot：

4.Expectation：
Please guide how to restore the daemonset csi-blob-node to running state in this case, and help analyze the cause of the problem, thank you very much.

### Tasks

[andyzhangx]

pls disable INSTALL_BLOBFUSE2 in the blob csi driver daemonset by running following command:

kubectl patch daemonset csi-blob-node -n kube-system -p '{"spec":{"template":{"spec":{"initContainers":[{"env":[{"name":"INSTALL_BLOBFUSE2","value":"false"},{"name":"BLOBFUSE2_VERSION","value":"2.3.0"}],"name":"install-blobfuse-proxy"}]}}}}'

[LeoandHenry]

Hi, @andyzhangx
Thank you for your reply, can I just use "kubectl edit daemonset csi-blob-node -n kube-system" to set INSTALL_BLOBFUSE2=false?

And could you explain the reason for doing it in this way? thank you!

[andyzhangx]

yes, and new node image has already installed blobfuse 2.3.0, that could be the reason why initContainer is crashing

[LeoandHenry]

Thank you very much @andyzhangx.
But when use above command "kubectl patch daemonset csi-blob-node -n kube-system -p '{"spec":{"template":{"spec":{"initContainers":[{"env":[{"name":"INSTALL_BLOBFUSE2","value":"false"},{"name":"BLOBFUSE2_VERSION","value":"2.3.0"}],"name":"install-blobfuse-proxy"}]}}}}'" the daemonset still has problems:

when I check the logs of the liveness-probe container, It shows that 29632 occupied. However, when I went to ont of the aks node, there were liveness probes that were used at that port, and even if I killed the process, it would restart immediately:

One more thing that needs to be synchronized with you: The csi-blob-node pod can start successfully after the customer uses the csi driver version 1.24.2 without setting INSTALL_BLOBFUSE2=false. I have tested this locally, so the customer wants to know why this is the case. Are there any changes in the 1.24.2 version?

[LeoandHenry]

By the way, in version 1.24.2, INSTALL_BLOBFUSE2 is true:

[andyzhangx]

@LeoandHenry INSTALL_BLOBFUSE2 should be set as false on aks node, while it should set as true on non-aks node since those nodes has not installed blobfuse2 by default.

[andyzhangx]

1.24.2 chart has blobfuse 2.3.0 installed by default, so v1.24.2 works

[LeoandHenry]

@andyzhangx Therefore, according to the previous discussion, the default version of blobfuse2 installed on the new node image is 2.3.0. After the customer upgrades the AKS node image, the customer csi driver is still of the old version, such as 1.23.3. Theoretically, the corresponding version of blobfuse2 is less than 2.3.0. However, in fact, the blobfuse2 of the node is already 2.3.0, and the old csi driver cannot match the new blobfuse2, which leads to the crash of initcontainer and the failure of csi-blob-node to start successfully.

[LeoandHenry]

Hi @andyzhangx ,If a newer version of csi driver such as 1.24.2 is installed, but the AKS node image is older and the blobfuse2 version on it is also old, such as 2.2.1, will csi-blob-node run properly?

[andyzhangx]

since aks node has already blobfuse2 binary installed, you should set --set node.blobfuseProxy.installBlobfuse2=false in helm chart install, so no matter what csi driver version you are going to install, it won't crash.

[LeoandHenry]

@andyzhangx thank you so much, I got it.