Unable to clone a repository on a CIFS share on Isilon storage. Error: rror: unable to append to 'X:/git/Win32-OpenSSH/.git/logs/refs/remotes/origin/HEAD': Invalid argument

[naatje80]

Unable to clone a repository on a CIFS share on Isilon storage.
Error: unable to append to 'X:/git/Win32-OpenSSH/.git/logs/refs/remotes/origin/HEAD': Invalid argument. This issue is only occurring in more recent version of Git for Windows.

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options
git version 2.14.1.windows.1
built from commit: 82d9b3f3b2407b52251620597d4b14933685459d
sizeof-long: 4
machine: x86_64

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver
Microsoft Windows [Version 10.0.14393]
(Windows 10 Enterprise 64-bit)

• What options did you set as part of the installation? Or did you choose the
  defaults?

> type "C:\Program Files\Git\etc\install-options.txt"
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

The issue seems to be related to:
#1262 (comment)
But I was asked to create a separate ticket for this because the issue there is caused by a rebase. I my case it is already caused by a clone.

In the related case I was asked to answer the following additional questions:
you have a Git repo hosted on disk (file system ext3, ext4,...?) on a Linux machine
OneFs 7.2.1.1 on EMC Isilon Cluster
the disk with the repo is made available on the network (via samba?)
EMC has their own protocol to serve CIFS.
you are accessing this samba share with Windows version ?
Windows 10 Enterprise (64-bit)

Can you run GIT_TRACE=1 GIT_CURL_VERBOSE=1 git $YOUR_COMMAND_THAT_FAILS and post the output here?

X:\git>git clone https://github.com/PowerShell/Win32-OpenSSH.git
10:46:42.634257 git.c:328               trace: built-in: git 'clone' 'https://github.com/PowerShell/Win32-OpenSSH.git'
Cloning into 'Win32-OpenSSH'...
10:46:43.853014 run-command.c:626       trace: run_command: 'remote-https' 'origin' 'https://github.com/PowerShell/Win32-OpenSSH.git'
10:46:43.915552 git.c:560               trace: exec: 'git-remote-https' 'origin' 'https://github.com/PowerShell/Win32-OpenSSH.git'
10:46:43.915552 run-command.c:626       trace: run_command: 'git-remote-https' 'origin' 'https://github.com/PowerShell/Win32-OpenSSH.git'
* Couldn't find host github.com in the _netrc file; using defaults
* timeout on name lookup is not supported
*   Trying 192.30.253.113...
* TCP_NODELAY set
* Connected to github.com (192.30.253.113) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=Delaware; serialNumber=5157550; street=88 Colin P Kelly, Jr Street; postalCode=94107; C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github.com
*  start date: Mar 10 00:00:00 2016 GMT
*  expire date: May 17 12:00:00 2018 GMT
*  subjectAltName: host "github.com" matched cert's "github.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
*  SSL certificate verify ok.
> GET /PowerShell/Win32-OpenSSH.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com
User-Agent: git/2.14.1.windows.1
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: BC46:5FA0:5A38BC:9C2B31:59F99804
< X-Frame-Options: DENY
<
* Connection #0 to host github.com left intact
10:46:44.759246 run-command.c:626       trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://github.com/PowerShell/Win32-OpenSSH.git/'
10:46:44.837370 git.c:328               trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://github.com/PowerShell/Win32-OpenSSH.git/'
* Couldn't find host github.com in the _netrc file; using defaults
* Found bundle for host github.com: 0x10a76b0 [can pipeline]
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (192.30.253.113) port 443 (#0)
> POST /PowerShell/Win32-OpenSSH.git/git-upload-pack HTTP/1.1
Host: github.com
User-Agent: git/2.14.1.windows.1
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 1002

* upload completely sent off: 1002 out of 1002 bytes
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: BC46:5FA0:5A3934:9C2B56:59F99804
< X-Frame-Options: DENY
<
remote: Counting objects: 7922, done.
remote: Compressing objects: 100% (13/13), done.
10:46:45.306136 run-command.c:626       trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 7572 on NLVEHVRES2NBNS1' '--check-self-contained-and-connected' '--pack_header=2,7922'
10:46:45.368619 git.c:328               trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 7572 on NLVEHVRES2NBNS1' '--check-self-contained-and-connected' '--pack_header=2,7922'
* Connection #0 to host github.com left intactB | 12.35 MiB/s
remote: Total 7922 (delta 3), reused 11 (delta 3), pack-reused 7906
Receiving objects: 100% (7922/7922), 134.75 MiB | 11.10 MiB/s, done.
Resolving deltas: 100% (5491/5491), done.
10:47:09.384103 run-command.c:626       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
10:47:09.462276 git.c:328               trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
error: unable to append to 'X:/git/Win32-OpenSSH/.git/logs/refs/remotes/origin/HEAD': Invalid argument
fatal: update_ref failed for ref 'HEAD': cannot update the ref 'HEAD': unable to append to 'X:/git/Win32-OpenSSH/.git/logs/HEAD': Invalid argument
cb0b7325a6ebe53f1ebcd19c67e30fc9a7683109.idx' failed. Should I try again? (y/n) n
Unlink of file 'Win32-OpenSSH/.git/objects/pack/pack-cb0b7325a6ebe53f1ebcd19c67e30fc9a7683109.idx' failed. Should I try again? (y/n) n

Some further information: I also setup a samba share on a Centos 7.4 workstation. The disk there is using ext4. On this machine it still seems to be working with git/2.14.1.windows.1.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Running using CMD and ToirtoiseGIT. Issue is the same.

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

See comments above.....

[naatje80]

The workaround mentioned in: #1262 (removing the "process = git-lfs filter-process") does not fix the issue in our case. The most recent version that still works is Git for Windows 2.12.2(2).

[dscho]

I bet that you can simplify the operation to something like git init followed by git commit -m empty --allow-empty, where the second command fails in the same way.

And you need to simplify it because you're the one stuck with debugging this further due to the pretty specific setup you got there.

Next step: follow the guidance given in https://github.com/git-for-windows/git/wiki/Debugging-Git to find out what call trace causes the error. Most likely you will end up around the places indicated here: https://github.com/git/git/search?utf8=%E2%9C%93&q=%22unable+to+append+to%22&type=Code meaning that probably some call in raceproof_create_file calls some function that has an emulation in compat/mingw.c whose GetLastError() -> errno translation encounters an unexpected condition. (it still might be a simple bug in that Isilon CIFS implementation).

[naatje80]

Ok, thank you. I understand that I'm the only person currently able to debug the issue. I will follow the mentioned steps an let you know the outcome.

[dscho]

@naatje80 any outcome yet?

[naatje80]

@dscho First of all, I'm very sorry for my delayed response. I finally got some time to further dive into the issue. You were indeed correct that the issue should be somewhere within raceproof_create_file. After debugging with gdb I finally found the call that was causing the issue. I've written a small c program to reproduce it:
`
#include "win32.h"
#include "fcntl.h"
#include <stdio.h>

int main()
{
HANDLE handle;

    DWORD create = (265 & ~O_CREAT) ? OPEN_ALWAYS : OPEN_EXISTING;

    LPCWSTR wfilename = L"./.git/history/test.txt";

   handle = CreateFileW(wfilename, FILE_APPEND_DATA,
                 FILE_SHARE_WRITE | FILE_SHARE_READ,
                 NULL, create, FILE_ATTRIBUTE_NORMAL, NULL);

    //errno = err_win_to_posix(GetLastError()), -1;
    errno = GetLastError(), -1;

    if ( handle == INVALID_HANDLE_VALUE ) {
            printf("ERROR, unable to create file\n");
            printf("Number: %d\n", errno);
    }

}
`
For this to fail, the directory .git/history should not exists. Normally an error ERROR_PATH_NOT_FOUND (3) should be returned. But when executing on the isilon_storage, a ERROR_INVALID_PARAMETER (87) is returned.
I found the following article which could be the cause for the issue:
https://support.microsoft.com/en-us/help/2990989/unexpected-errors-are-returned-when-a-shared-folder-is-accessed-from-a
But I'm not 100% sure this is the way it should be solved for Git for Windows.

I could indeed resolve the issue in the sources, by changing the following line of code in compat/mingw.c for the error translation:
case ERROR_INVALID_PARAMETER: error = EINVAL; break;
to:
case ERROR_INVALID_PARAMETER: error = ENOENT; break;

[dscho]

I could indeed resolve the issue in the sources, by changing the following line of code in compat/mingw.c for the error translation:
case ERROR_INVALID_PARAMETER: error = EINVAL; break;
to:
case ERROR_INVALID_PARAMETER: error = ENOENT; break;

That would probably break other users of that error translation. But I think we could special-case this in the caller (is it mingw_open()?), i.e. something like this:

DWORD err = GetLastError();
if (err == ERROR_INVALID_PARAMETER)
    err = ERROR_FILE_NOT_FOUND;
errno = err_win_to_posix(err);

Could you run with that and give it a try?

[naatje80]

I could indeed resolve the issue in the sources, by changing the following line of code in compat/mingw.c for the error translation:
case ERROR_INVALID_PARAMETER: error = EINVAL; break;
to:
case ERROR_INVALID_PARAMETER: error = ENOENT; break;

That would probably break other users of that error translation.

I was already afraid of that :)

But I think we could special-case this in the caller (is it mingw_open()?), i.e. something like this:

DWORD err = GetLastError();
if (err == ERROR_INVALID_PARAMETER)
    err = ERROR_FILE_NOT_FOUND;
errno = err_win_to_posix(err);

Could you run with that and give it a try?

I think it was indeed in mingw_open. I will add this and let you know the outcome.

[naatje80]

In this case in was mingw_open_append, I've implemented it in the following way:

   if (handle == INVALID_HANDLE_VALUE) {
                DWORD err = GetLastError();
                if (err == ERROR_INVALID_PARAMETER)
                        err = ERROR_FILE_NOT_FOUND;
                return errno = err_win_to_posix(err), -1;
        }

This indeed fixed the issue. But I still need to check if there are no other actions that are now blocked, because I assume this is only used when a file needs to be appended. Again, I will let you know.

Thank you for all your help!

[dscho]

Excellent progress!

[naatje80]

I've also test some pull and clone actions, but the issue for those actions was also resolved. I assume your suggestion should fix the issue.

[dscho]

Nice. Can you wrap this up in a neat PullRequest for me to review and merge?

[naatje80]

Sure, I will create a pull request for the required changes. Thank you again for your help so far.