tmp_objdir-incoming directory not empty on push #3866
Description
Since we upgraded our file server from Debian stretch (9) to bullseye (11), pushing to the server often gives errors. This is an upgrade from samba 4.5.16 to 4.13.13. Since we saw the issue, we've tried upgrading to version 2.34.1 in January, but we're still seeing the issue. We're using the 64 bit version of git bash running on Windows 10.
A session looks like this:
$ git push server master:master
Enumerating objects: 70, done.
Counting objects: 100% (70/70), done.
Delta compression using up to 6 threads
Compressing objects: 100% (37/37), done.
Writing objects: 100% (37/37), 7.87 MiB | 1004.00 KiB/s, done.
Total 37 (delta 31), reused 0 (delta 0), pack-reused 0
warning: unable to unlink './objects/tmp_objdir-incoming-a15328/29': Directory not empty
error: unable to write file ./objects/29: Directory not empty
warning: unable to unlink './objects/tmp_objdir-incoming-a15328/65': Directory not empty
error: unable to write file ./objects/65: Directory not empty
warning: unable to unlink './objects/tmp_objdir-incoming-a15328/6c': Directory not empty
error: unable to write file ./objects/6c: Directory not empty
warning: unable to unlink './objects/tmp_objdir-incoming-a15328/6d': Directory not empty
error: unable to write file ./objects/6d: Directory not empty
warning: unable to unlink './objects/tmp_objdir-incoming-a15328/7e': Directory not empty
error: unable to write file ./objects/7e: Directory not empty
warning: unable to unlink './objects/tmp_objdir-incoming-a15328/8f': Directory not empty
error: unable to write file ./objects/8f: Directory not empty
warning: unable to unlink './objects/tmp_objdir-incoming-a15328/98': Directory not empty
error: unable to write file ./objects/98: Directory not empty
warning: unable to unlink './objects/tmp_objdir-incoming-a15328/a7': Directory not empty
error: unable to write file ./objects/a7: Directory not empty
warning: unable to unlink './objects/tmp_objdir-incoming-a15328/c2': Directory not empty
error: unable to write file ./objects/c2: Directory not empty
To file:///s/<PATH>/repo.git
! [remote rejected] master -> master (unable to migrate objects to permanent storage)
error: failed to push some refs to 'file:///s/<PATH>/repo.git'
Looking on the server, the tmp_objdir-incoming-a15328 still exists, all the mentioned directories exist, and they contain at least 1 file. Other directories also still exist but are empty.
Retrying the command gives this as result:
$ git push server master:master
Enumerating objects: 70, done.
Counting objects: 100% (70/70), done.
Delta compression using up to 6 threads
Compressing objects: 100% (37/37), done.
Writing objects: 100% (37/37), 7.87 MiB | 2.14 MiB/s, done.
Total 37 (delta 31), reused 0 (delta 0), pack-reused 0
warning: unable to unlink './objects/tmp_objdir-incoming-a08948/29': Directory not empty
error: unable to write file ./objects/29: Directory not empty
warning: unable to unlink './objects/tmp_objdir-incoming-a08948/7e': Directory not empty
error: unable to write file ./objects/7e: Directory not empty
warning: unable to unlink './objects/tmp_objdir-incoming-a08948/a7': Directory not empty
error: unable to write file ./objects/a7: Directory not empty
To file:///s/<PATH>/repo.git
! [remote rejected] master -> master (unable to migrate objects to permanent storage)
error: failed to push some refs to 'file:///s/<PATH>/repo.git'
Retrying again gives:
git push server master:master
Enumerating objects: 70, done.
Counting objects: 100% (70/70), done.
Delta compression using up to 6 threads
Compressing objects: 100% (37/37), done.
Writing objects: 100% (37/37), 7.87 MiB | 2.58 MiB/s, done.
Total 37 (delta 31), reused 0 (delta 0), pack-reused 0
warning: unable to unlink './objects/tmp_objdir-incoming-a14780/a7': Directory not empty
error: unable to write file ./objects/a7: Directory not empty
To file:///s/<PATH>/repo.git
! [remote rejected] master -> master (unable to migrate objects to permanent storage)
error: failed to push some refs to 'file:///s/<PATH>/repo.git'
That is, less directories are non-empty after a retry. After trying it enough times it's successful. The next time we try and we still get an error, it's still one of the same directories as the previous attempt. In that session we had the error with only a7 being non-empty 5 times.