Skip to content

Conversation

andyzhangx
Copy link
Member

@andyzhangx andyzhangx commented Mar 17, 2023

What type of PR is this?
/kind bug

What this PR does / why we need it:
fix: add workaround for blobfuse2 mount issue
unlike blobfusev1, blobfuse2 would create a new child process to handle mount, thus blobfuse2 mount becomes async now, there would be race condition between real mount process and next bind mount operation if it's async mount,
this PR would sleep 2s waiting for mount process complete
details: Azure/azure-storage-fuse#1079

Which issue(s) this PR fixes:

Fixes #

Requirements:

Special notes for your reviewer:

  • verified manually, after sleep 2s, there is no more mount on local disk any more:
# mount | grep blobfuse2
blobfuse2 on /var/lib/kubelet/plugins/kubernetes.io/csi/blob.csi.azure.com/1f778bb6414796cfefe77ad67ee6e5f21be127ed1adf86affb766488a15ebee4/globalmount type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
blobfuse2 on /var/lib/kubelet/pods/5a555547-46b4-45f3-8da6-4896903a8dd3/volumes/kubernetes.io~csi/pvc-32e09edb-ac53-4240-976a-6e810e270018/mount type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
root@aks-agentpool-20541019-vmss000000:/# mount | grep sda
/dev/sda1 on / type ext4 (rw,relatime,discard)
/dev/sda15 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/sda1 on /var/lib/kubelet type ext4 (rw,relatime,discard)
  • also there is no device or resource busy during unmount now

Release note:

fix: add workaround for blobfuse2 mount race condition issue

@k8s-ci-robot k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 17, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 17, 2023
@andyzhangx
Copy link
Member Author

/assign @cvvz

@andyzhangx
Copy link
Member Author

/retest

@andyzhangx
Copy link
Member Author

/cherrypick release-1.20
/cherrypick release-1.19

@k8s-infra-cherrypick-robot

@andyzhangx: once the present PR merges, I will cherry-pick it on top of release-1.20 in a new PR and assign it to you.

In response to this:

/cherrypick release-1.20
/cherrypick release-1.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@andyzhangx
Copy link
Member Author

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 18, 2023
@andyzhangx andyzhangx force-pushed the fuse2-mount-workaround branch from 3bc1c12 to fa9f8a5 Compare March 18, 2023 01:25
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 18, 2023
@k8s-ci-robot
Copy link
Contributor

@andyzhangx: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-blob-csi-driver-e2e-capz fa9f8a5 link true /test pull-blob-csi-driver-e2e-capz

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@andyzhangx andyzhangx removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 18, 2023
@andyzhangx andyzhangx changed the title fix: add workaround for blobfuse2 mount issue fix: add workaround for blobfuse2 mount race condition issue Mar 18, 2023
@andyzhangx andyzhangx merged commit 709cd20 into kubernetes-sigs:master Mar 18, 2023
@k8s-infra-cherrypick-robot

@andyzhangx: cannot checkout release-1.20: error checking out release-1.20: exit status 1. output: error: pathspec 'release-1.20' did not match any file(s) known to git

In response to this:

/cherrypick release-1.20
/cherrypick release-1.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-infra-cherrypick-robot

@andyzhangx: new pull request created: #843

In response to this:

/cherrypick release-1.20
/cherrypick release-1.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@andyzhangx
Copy link
Member Author

/cherrypick release-1.20

@k8s-infra-cherrypick-robot

@andyzhangx: new pull request created: #844

In response to this:

/cherrypick release-1.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants