Unable to clone repositories in Windows Containers

[atrauzzi]

Cross posting from here: microsoft/dotnet-framework-docker#61

Not sure what happened, but up until last week, I was able to clone git repositories via SSH-based URIs in my Windows containers. Now, suddenly without warning, when I go to clone - even a public repository, I get this error:

C:\>git clone [email protected]:atrauzzi/praxis.git
git clone [email protected]:atrauzzi/praxis.git
Cloning into 'praxis'...
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Here's a simple Dockerfile that can be used to reproduce the issue:

FROM microsoft/dotnet-framework:4.7.1-windowsservercore-1709
SHELL ["powershell"]

RUN Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
RUN choco install --yes git

RUN refreshenv

I'm not sure at this point what might be the issue - whether it's Windows or git. The only suspicious thing is that if I try to run the mingw based ssh.exe, it prints a blank line and then exits. My gut is telling me something about the SSH support in git for Windows is having a hard time?

[atrauzzi]

So. upon doing some more digging, it looks like the issue is to do with known_hosts not having the key for github.com. If I manually enter the container and run this command, I'm able to clone repositories:

./ssh-keyscan -v -H github.com >> /Users/ContainerAdministrator/.ssh/known_hosts

Unfortunately, as part of my container automation, I'm faced with some difficulties in resolving this correctly. My initial instinct was to add this to my Dockerfile:

WORKDIR '/Program Files/Git/usr/bin'
RUN ./ssh-keyscan -v -H github.com >> /Users/ContainerAdministrator/.ssh/known_hosts

Which gives me this error message during my container build:

Step 8/11 : RUN ./ssh-keyscan -v -H github.com >> /Users/ContainerAdministrator/.ssh/known_hosts
 ---> Running in 5132a8ee3d63
The command 'powershell ./ssh-keyscan -v -H github.com >> /Users/ContainerAdministrator/.ssh/known_hosts' returned a non-zero code: 1

So now I'm left trying to figure out what about being run in a container build seems to be upsetting the SSH tools bundled with git for Windows. I'm also a bit frustrated that I've had no error feedback at any step along the way.

If I try to run it via cmd rather than powershell, I get this error:

The command 'cmd /C "ssh-keyscan -v -H github.com" > /known_hosts' returned a non-zero code: 3221225794

[atrauzzi]

cc. docker/for-win#1351

[dscho]

it looks like the issue is to do with known_hosts not having the key for github.com

Not that I claim to know how to resolve this properly, but maybe you want to imitate what I do in my VSTS jobs:

mkdir -p .ssh &&
echo 'github.com,192.30.252.128 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==' >>.ssh/known_hosts

Sure, this would fail if the host key changed. But that's exactly what I would want it to do: to fail, so that I can figure out whether it was really GitHub that changed the key, or whether it was my cute buddy playin' a trick on me.

[atrauzzi]

@dscho - Indeed, I think I'm going to have to do something like that in the interim. It would be nice if I could have my Dockerfile pull that information as per the keyscan.

Thanks for the snippet, I'll probably end up using it until this can be fixed properly.

[thecloudtaylor]

Two part solution...
Part 1: Use the OpenSSH port for Windows (which is going in box!)
Part 2: Pre-create the .ssh directory

Part 1

FROM microsoft/windowsservercore:1709

ENV OPENSSH_VERSION 0.0.24.0
ENV OPENSSH_DOWNLOAD_URL "https://github.com/PowerShell/Win32-OpenSSH/releases/download/$OPENSSH_VERSION/OpenSSH-Win64.zip"

RUN powershell -Command
$ErrorActionPreference = 'Stop';
(New-Object System.Net.WebClient).DownloadFile('%OPENSSH_DOWNLOAD_URL%', 'openssh.zip') ;
Expand-Archive openssh.zip -DestinationPath c:\ ;
Remove-Item openssh.zip -Force

RUN setx PATH %PATH%;c:\OpenSSH-Win64

Part 2

FROM openssh

RUN mkdir C:\Users\ContainerAdministrator\.ssh
RUN ssh-keyscan.exe -v -H github.com >> /Users/ContainerAdministrator/.ssh/known_hosts

[Sean-Brown]

@taylorb-microsoft
I'm using a Bitbucket server, not Github, but the principal should be the same right? It doesn't seem to work for me though.

After installing openssh and adding the server's key to the known_hosts file (tested by successfully opening an ssh session on the Bitbucket server), I copied my Bitbucket ssh key to C:\Users\ContainerAdministrator\.ssh\, but now I still can't get the container to clone my repo.

Running ssh-agent doesn't output anything, but I believe it is running because ssh-add works:

Identity added: C:\Users\ContainerAdministrator.ssh\id_rsa (C:\Users\ContainerAdministrator.ssh\id_rsa)

but still cloning does not work:

Cloning into 'repo'...
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I'm curious if there are additional steps required to get this to work? Have you actually been able to successfully clone a Git repo in a Windows container?

[dscho]

Have you actually been able to successfully clone a Git repo in a Windows container?

Yes ;-) Almost all of Git for Windows' continuous testing and integration is performed in Docker containers itself.

BTW did you eval the output of ssh-agent? Otherwise, subsequent ssh sessions won't know how to connect to the agent.

To diagnose your woes further, I would suggest running ssh directly with the host name, with a liberal amount of -v thrown in: ssh -v -v -v -v -v -v bitbucket.org.

[Sean-Brown]

BTW did you eval the output of ssh-agent?

Is there an add-in that brings this command to powershell? Otherwise I don't seem to have eval as a recognized cmdlet.

Thanks for the -v pointer, it gives me a good starting point for debugging.

[Sean-Brown]

Thanks for the assistance, using that verbose flag helped pinpoint the issue: I was using a config file in my user's .ssh directory but apparently I needed to remove some permissions for a couple of users, as indicated by an error message:

debug3: Bad permissions. Try removing permissions for user: S-1-5-11 on file C:\Users\ContainerAdministrator/.ssh/config.

[dscho]

@atrauzzi can this ticket be closed?

[Custodia]

I'm having a similar problem but I haven't been able to pinpoint the problem yet. I just started looking into this and I don't seem to have the same Bad permissions error message. I do however have multiple errors like this: debug3: Failed to open file:C:\\Users\\ContainerAdministrator/.ssh/id_dsa error:2 for all the ssh files I've copied over to the container.

Did you use git clone without ssh private/public keys? And what user where you able to run the clone commands with? @atrauzzi @Sean-Brown

[Custodia]

Scratch what I said above, with ssh -vvvvvv mybitbucket.com I'm able to get Authenticated to mybitbucket.com ([192.168.70.1]:22). however git clone still fails with the same non descriptive error.

[dscho]

Failed to open file:C:\Users\ContainerAdministrator/.ssh/id_dsa error:2

The file: part looks dubious. Maybe your HOME variable is a bit funny?

[Sean-Brown]

I've been experimenting with this in a container and I thought I had it working but now I'm unable to reproduce the container -- I am always getting git clone failures, which is confusing because I thought I was doing the same procedure but now it doesn't work and git clone, even with trace enabled, is very non-descriptive. Not only that, but redirecting the ssh output into a file does not work whatsoever in Windows. @Custodia I'll let you know if I can reliably get it to work.

[thecloudtaylor]

Ok I did some more digging and found another issue but also a workaround. It seems like git is calling the version of SSH it installed vs OpenSSH. Below is a complete dockerfile the installs git and OpenSSH. It also registers the ssh services, sets the ssh-agent service to auto start and does the hack of copying the OpenSSH binaries over the top of the git installed version :/.

After all this here's the test I was able to successfully complete:

docker build -t git .
docker run -it git cmd
##in the container now##
ssh-keygen -t rsa -b 4096 -C "<email>"  ##accepting defaults##
ssh-add C:\Users\ContainerAdministrator\.ssh\id_rsa
type C:\Users\ContainerAdministrator\.ssh\id_rsa.pub ##adding key to GitHub##

git clone [email protected]:Microsoft/hcsshim.git

Dockerfile:

FROM microsoft/windowsservercore:1709

#ENV
ENV CHOCOLATEY_INSTALL "https://chocolatey.org/install.ps1"
ENV OPENSSH_VERSION 0.0.24.0
ENV OPENSSH_DOWNLOAD_URL "https://github.com/PowerShell/Win32-OpenSSH/releases/download/$OPENSSH_VERSION/OpenSSH-Win64.zip"
ENV OPENSSH_INSTALL "c:\OpenSSH-Win64"

#Install GitTools
RUN powershell -Command \
    Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('%CHOCOLATEY_INSTALL%'))

RUN choco install --yes git

RUN refreshenv

#Install OpenSSH
RUN powershell -Command \
    $ErrorActionPreference = 'Stop'; \ 
    (New-Object System.Net.WebClient).DownloadFile('%OPENSSH_DOWNLOAD_URL%', 'openssh.zip') ; \
    Expand-Archive openssh.zip -DestinationPath c:\ ; \
    Remove-Item openssh.zip -Force
#Installing Services (per https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH)
RUN powershell -ExecutionPolicy Bypass -File %OPENSSH_INSTALL%\install-sshd.ps1
RUN sc config ssh-agent start=auto
#Fixing premissions (per https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH)
RUN Powershell.exe -ExecutionPolicy Bypass -Command ". %OPENSSH_INSTALL%\FixHostFilePermissions.ps1 -Confirm:$false"
#Adding SSH to Path
RUN setx PATH "%OPENSSH_INSTALL%;%PATH%"

#HACK, overrids the SSH version that comes with GIT
RUN copy %OPENSSH_INSTALL%\ssh*.exe "c:\Program Files\Git\usr\bin" /y

[Sean-Brown]

Oh I didn't know that, btw it looks like there is a git environment variable for telling it which ssh.exe to use, in powershell it'd be something like: [System.Environment]::SetEnvironmentVariable('GIT_SSH', 'C:\Program Files\OpenSSH-Win64\ssh.exe', 'Machine')
I just tried this however and it didn't solve the issue for me, but maybe I didn't do it correctly?

[Sean-Brown]

OK I tried over-writing the ssh executables like you said and I was able to clone the repo, however getting submodules is somehow broken, i.e. running git submodule update --init --recursive doesn't pull the submodules and nothing is output from that command.

UPDATE: I recreated the container, but this time setting GIT_SSH to point to the OpenSSH version of ssh.exe works: I am able to clone the repo. I still am unable to get the submodules however.

[thecloudtaylor]

I wasn't getting GIT_SSH working either, I suspect but I'm not positive that while GIT_SSH points to ssh.exe but there are other binaries that might be in conflict...

I'm not sure why git submodule isn't working, though I suspect its similar to the root problem of ssh (something to do with msys).

[Custodia]

@taylorb-microsoft your hack fix worked for me aswell.

[Sean-Brown]

I believe the issue with git submodule is that it's trying to run a shell script, and apparently I haven't installed something right because sh.exe doesn't produce any output. I have been able to run git submodule--helper update-clone and this command does give me feedback and is able to query the status of those submodules successfully: I'm just trying to find what the equivalent of git submodule update --init --recursive translates to in terms of git submodule--helper update-clone. I'm looking at git-submodule.sh but it's not apparent what the translation would be.

[dscho]

sh.exe doesn't produce any output

There is likely some sh.exe.stackdump lying around afterwards :-(

I'm just trying to find what the equivalent of git submodule update --init --recursive translates to in terms of git submodule--helper update-clone.

I fear that it does not do that. The helper only handles one submodule at a time. The recursion is what the git rebase Unix shell script handles.

As I suggested earlier, if you have problems with shell scripts in Docker containers, maybe using the -BusyBox variant of MinGit helps? (You only need MinGit, anyway, as you do not plan on using fancy UIs in Docker, right?)

[dscho]

maybe using the -BusyBox variant of MinGit helps?

I just realized that I suggested this not only earlier, but elsewhere.

So please let me clarify: MinGit is a subset of Git for Windows, intended for applications. All purely interactive commands have been stripped, and also git svn is excluded (for size reasons). It should be enough for any scripted usage, including Docker containers.

MinGit is released simultaneously with every Git for Windows version.

There is a new, experimental kid on the block: BusyBox-based MinGit. It not only shrinks size, but it also avoids the Cygwin derivative MSYS2 in many cases (although not for OpenSSH). The BusyBox component has still not seen any serious battle testing, that is the entire reason why MinGit still uses MSYS2. The BusyBox based MinGit can be found here: https://github.com/git-for-windows/git/releases/download/v2.15.1.windows.2/MinGit-2.15.1.2-busybox-64-bit.zip

Your git submodule problems may have their root cause in that MSYS2 part that runs the Unix shell scripts, and BusyBox should fare much better executing them.

Furthermore, MinGit (both the regular variant as well as the BusyBox one) are distributed as plain .zip files, so it should be super easy to extract via Powershell:

$WebClient = New-Object System.Net.WebClient; $WebClient.DownloadFile("https://github.com/git-for-windows/git/releases/download/v2.15.1.windows.2/MinGit-2.15.1.2-64-bit.zip", "MinGit-2.15.1.2-64-bit.zip")
mkdir MinGit | Out-Null
Add-Type -AssemblyName System.IO.Compression.FileSystem ; [System.IO.Compression.ZipFile]::ExtractToDirectory("MinGit-2.15.1.2-64-bit.zip", "MinGit")

[thecloudtaylor]

@dscho it looks like sh (or really bash.exe) is AV'ing (it's producing a memory dump with an AV is msys someplace)…

I created a dockerfile combining OpenSSH and MinGit - I was able to clone the repro still and preform basic commands. git submodule is at least returning output with -? so it might work.

FROM microsoft/windowsservercore:1709

#ENV
ENV MINGET_DOWNLOAD_URL "https://github.com/git-for-windows/git/releases/download/v2.15.1.windows.2/MinGit-2.15.1.2-busybox-64-bit.zip"
ENV OPENSSH_VERSION 0.0.24.0
ENV OPENSSH_DOWNLOAD_URL "https://github.com/PowerShell/Win32-OpenSSH/releases/download/$OPENSSH_VERSION/OpenSSH-Win64.zip"
ENV OPENSSH_INSTALL "c:\OpenSSH-Win64"

#Install MinGit
RUN powershell -Command \
    $ErrorActionPreference = 'Stop'; \ 
    (New-Object System.Net.WebClient).DownloadFile('%MINGET_DOWNLOAD_URL%', 'mingit.zip') ; \
    Expand-Archive mingit.zip -DestinationPath c:\mingit ; \
    Remove-Item mingit.zip -Force

RUN setx PATH "c:\mingit\cmd;%PATH%"

#Install OpenSSH
RUN powershell -Command \
    $ErrorActionPreference = 'Stop'; \ 
    (New-Object System.Net.WebClient).DownloadFile('%OPENSSH_DOWNLOAD_URL%', 'openssh.zip') ; \
    Expand-Archive openssh.zip -DestinationPath c:\ ; \
    Remove-Item openssh.zip -Force
#Installing Services (per https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH)
RUN powershell -ExecutionPolicy Bypass -File %OPENSSH_INSTALL%\install-sshd.ps1
RUN sc config ssh-agent start=auto
#Fixing premissions (per https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH)
RUN Powershell.exe -ExecutionPolicy Bypass -Command ". %OPENSSH_INSTALL%\FixHostFilePermissions.ps1 -Confirm:$false"
#Adding SSH to Path
RUN setx PATH "%OPENSSH_INSTALL%;%PATH%"
ENV GIT_SSH "c:\OpenSSH-Win64\ssh.exe"

[dscho]

I was able to clone the repro still and preform basic commands. git submodule is at least returning output with -? so it might work.

So you're saying this fixes the described issues in your setup? If so: great!

[Sean-Brown]

I'll give BusyBox MinGit a shot, if it works I'll favor it over my hack using git submodule--helper, which I got to work for pulling submodules by:

1. running git submodule--helper update-clone --update checkout
2. cd into each submodule directory and run git reset --hard, since for some reason I can't get that update-clone command to actually pull down the changes (Git thinks I want to delete the entire contents of those submodules)

UPDATE: BusyBox MinGit works perfectly :) Thanks for your assistance on this issue, it was driving me crazy, glad to get it resolved

[dscho]

UPDATE: BusyBox MinGit works perfectly :) Thanks for your assistance on this issue, it was driving me crazy, glad to get it resolved

Thanks for your feedback! I will try to set aside time (hopefully in the next sprint) to try to get BusyBox-based MinGit further along.

[Custodia]

My problem now is that I'm trying to use the machine as a Jenkins slave. It works perfectly fine if I add the ssh keys to the image manually but if I try to pass the keys from the master the build will fail because of the OpenSSH permission problems on windows. Doesn't sound like using mingit would help either since it's using the same version of ssh as the regular git installation.

Git was last updated in chocolatey November 30th and atrauzzi mentioned that cloning has worked previously, I wonder if using Git 2.15.0 instead of Git 2.15.1.2 would work. I'm going to try that next.

edit: Cloning does not work with 2.15.0

[thecloudtaylor]

This line RUN Powershell.exe -ExecutionPolicy Bypass -Command ". %OPENSSH_INSTALL%\FixHostFilePermissions.ps1 should fix the prems on the keys

[Custodia]

@taylorb-microsoft That doesn't help since Jenkins generates the key files on demand into temporary folders. I also can't add that to any running builds because cloning happens (and fails) before anything else. I'm reverting to https for now because I don't want to bake the ssh perm files into the docker images. But the solution did work perfectly fine when I tried that.

[thecloudtaylor]

@Custodia do you have any control over where the keys are placed? i.e. we might be able to fix this via the right inheritance settings on the root directory.

[Custodia]

The keys are stored in C:\\Build\\Workspace\\BuildName@tmp\\sshXXXXXXXXXXXXXX.key, so you do know where they are stored but you would have to predict the build names when building the docker containers which is also too much of a hassle compared to just using https.

[aL3891]

Old issue but I stumbled across it and it helped me end up with this docker file. It uses a supplied .ssh folder instead of generating a key:

FROM microsoft/windowsservercore

ADD .ssh /Users/ContainerAdministrator/.ssh

ADD https://github.com/git-for-windows/git/releases/download/v2.17.0.windows.1/MinGit-2.17.0-busybox-64-bit.zip mingit.zip
ADD https://github.com/PowerShell/Win32-OpenSSH/releases/download/v7.6.1.0p1-Beta/OpenSSH-Win64.zip openssh.zip

RUN powershell Expand-Archive mingit.zip -DestinationPath c:\mingit;
RUN powershell Expand-Archive openssh.zip -DestinationPath c:\;

RUN del mingit.zip
RUN del openssh.zip

RUN setx PATH "%PATH%;c:\mingit\cmd;c:\OpenSSH-Win64"
RUN powershell -File c:\OpenSSH-Win64\install-sshd.ps1
RUN sc config ssh-agent start=auto
RUN Powershell.exe -File c:\OpenSSH-Win64\FixHostFilePermissions.ps1
ENV GIT_SSH "c:\OpenSSH-Win64\ssh.exe"```

[csteenwyk]

I struggled with this for a while as well. I used the above comments to install OpenSSH into my windows image but it still didn't work. From another post I found that there is an environment variable, GIT_SSH, which tells git where ssh.exe is. Once I set that, everything worked great. So I added this to my dockerfile:

ADD http://github.com/PowerShell/Win32-OpenSSH/releases/download/0.0.24.0/OpenSSH-Win64.zip C:\TEMP\openssh.zip
RUN powershell -Command $ErrorActionPreference = 'Stop'; Expand-Archive C:\TEMP\openssh.zip -DestinationPath c:\ 
RUN setx PATH %PATH%;c:\OpenSSH-Win64
ENV GIT_SSH C:\OpenSSH-Win64\ssh.exe

[dscho]

I'll simply interpret the latest comment as the workaround everybody was waiting for.