Investigate `git daemon` problems

[dscho]

It had been reported in msysgit#70 (with a quite large patch that I did not manage to review due to its sheer size) that git daemon might close the socket before the client read everything, and as a consequence, the transaction fails.

I still need to be able to reproduce; if I manage that, I will try to come up with a substantially smaller patch myself.

[linquize]

Not easy to report a MCVE, because after a few times of retry, it succeeds.
But in some cases, it still fails with 100 consecutive times of retry.

[wq9]

I always get this problem.

Client (git version 2.5.0.windows.1):

Server (git version 2.4.6.windows.1):

C:\vpk>git daemon --verbose --base-path=C:\
[2652] Ready to rumble
[3296] Connection from <client>:59471
[3296] Extended attributes (32 bytes) exist <host=vm1-kykbkmi4.cloudapp.net>
[3296] Request upload-pack for '/vpk/repo'
[2652] [3296] Disconnected
[1776] Connection from <client>:59486
[1776] Extended attributes (32 bytes) exist <host=vm1-kykbkmi4.cloudapp.net>
[1776] Request upload-pack for '/vpk/repo'
[2652] [1776] Disconnected

[dscho]

@wq9 the trick, now, is to put in some wait cycles at the appropriate point to trigger this always (i.e. also here)... ;-)

[Radrik5]

I also get this problem when I try fetching from remote machine. I cannot reproduce it using localhost (it works fine) but for remote machine git fetch git:// always fails. With Git 1.9.5 it worked well in most cases (only sometimes it failed).

Remote machine: Windows 7 64bit, Git 2.7.0 64bit
Local machine: Windows 7 64bit, Git 2.7.2 64bit

Command on remote machine:

git daemon --base-path=. --export-all --reuseaddr --informative-errors --verbose

Command on local machine:

$ GIT_TRACE=1 git fetch git://host/ test:host
13:27:40.171077 git.c:348               trace: built-in: git 'fetch' 'git://host/' 'test:host'
13:27:40.366077 run-command.c:343       trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
remote: Counting objects: 65, done.
remote: Compressing objects: 100% (65/65), done.
13:27:40.759077 run-command.c:343       trace: run_command: 'unpack-objects' '--pack_header=2,65'
13:27:40.782077 git.c:348               trace: built-in: git 'unpack-objects' '--pack_header=2,65'
fatal: read error: Invalid argument
fatal: early EOFs:  23% (15/65)
fatal: unpack-objects failed

I think you can try any repo on remote machine and git fetch should fail. The point here is not to use localhost network.

[linquize]

Still a problem in 2.7

[guban]

I prepared the following MCVE:

My setup is

Windows 10 64-bit.

Git-2.11.0.3-64-bit.exe

$ git --version
git version 2.11.0.windows.3

The problem is reproducible when cloning from a local machine. To
reproduce the problem, first run the following script as a batch file so
as to generate a git repository and start git daemon:

set work_dir=%cd%
set repo_dir=%work_dir%\my-repo

:: Uncomment this for cleaning-up after previous run.
::rmdir /s /q %repo_dir%

mkdir %repo_dir%
cd %repo_dir%
git init
echo 1 > .git\git-daemon-export-ok

:: This populates a.txt with the help text for the "for" command.
help for > a.txt

:: Can't reproduce without generating quite a bit of history.
:: Tweak the last number in the triplet to vary the history size.
:: Beware of exponential growth. Value 12 results in a 64M repository.
for /l %%x in (1, 1, 12) do (
  type a.txt >> a.txt
  git add a.txt
  git commit -m "update %%x"
)
set GIT_TRACE=1
git daemon --verbose --reuseaddr --base-path=%work_dir:\=/% %work_dir:\=/%

Then, try cloning this repository:

set GIT_TRACE=1
git clone git://127.0.0.1/my-repo cloned

Most of the time (but not always), the above git clone command fails:

09:33:24.142211 git.c:371               trace: built-in: git 'clone' 'git://127.0.0.1/my-repo' 'cloned'
09:33:24.201829 run-command.c:350       trace: run_command: 'index-pack' '--stdin' '--fix-thin' '--keep=fetch-pack 8892 on oleg' '--check-self-contained-and-connected'
Cloning into 'cloned'...
09:33:24.210832 git.c:371               trace: built-in: git 'index-pack' '--stdin' '--fix-thin' '--keep=fetch-pack 8892 on oleg' '--check-self-contained-and-connected'
remote: 09:33:24.210832 git.c:371               trace: built-in: git 'pack-objects' '--revs' '--thin' '--stdout' '--delta-base-offset'        
fatal: early EOF
fatal: read error: Invalid argument
fatal: index-pack failed

The corresponding git daemon output is

[7296] Connection from 127.0.0.1:26663
[7296] unable to set SO_KEEPALIVE on socket: No such file or directory
[7296] Extended attributes (16 bytes) exist <host=127.0.0.1>
[7296] Request upload-pack for '/my-repo'
09:33:24.175817 run-command.c:350       trace: run_command: 'upload-pack' '--strict' '--timeout=0' '.'
[7296] 09:33:24.185821 git.c:607               trace: exec: 'git-upload-pack' '--strict' '--timeout=0' '.'
[7296] 09:33:24.185821 run-command.c:350       trace: run_command: 'git-upload-pack' '--strict' '--timeout=0' '.'
[7296] 09:33:24.199827 run-command.c:350       trace: run_command: 'pack-objects' '--revs' '--thin' '--stdout' '--delta-
base-offset'

Sometimes, git clone fails with another error:

fatal: early EOFs: 69% (53/76)
fatal: read error: Invalid argument
fatal: index-pack failed

I also observe the following surprising behavior. When I select any text
in the console window where git daemon is running, subsequent cloning
operations always succeed. Then, when I clear the selection with Escape
key, git clone begins to fail intermittently again. Apparently, the
text selection suppresses git daemon's console output, and that somehow
helps to avoid the condition that leads to git clone failure.

The problem is also reproducible in 32-bit version 2.11.0.windows.3.

[cezariuszmarek]

It's 100% reproducible on my 6 MB repository with 183 history entries. Can't you remove the mcve-required label having @guban script?

[guban]

If there is a reason for keeping this issue as "mcve-required", it would be nice to let me know what needs to be improved in my MCVE that was provided back in January.

[dscho]

I did not yet have time to come back to this issue. So I am reluctant to remove that tag, especially given that no contributor who would be able to fix the underlying problem gave the MCVE a whirl. Sorry, busy times!

[edropps]

We have a team of ~ 10 people who run into this exact issue daily on windows Server 2016 on multiple Git versions through Git-2.16.1.3-64-bit I support removing the mcve-required tag.