Credential Manager doesn't work in Windows Containers with BusyBox MinGit

[dhirschfeld]

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

C:\> git --version --build-options
git version 2.16.1.windows.4
cpu: x86_64
built from commit: ef6d451bbfef86a529ebf12620289e0f15a93d8e

Running on a windowsservercore:1709 container:

C:\> (Get-CimInstance Win32_OperatingSystem).version
10.0.16299

C:\> cmd.exe /c ver

Microsoft Windows [Version 10.0.16299.192]

• What options did you set as part of the installation? Or did you choose the
  defaults?

I just unzipped the prebuilt release

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Running on a Windows Container

Details

• Running from powershell

C:\> git clone https://github.stanwell.com/Trading/skeleton.git
Cloning into 'skeleton'...
bash: Unhandled device /dev/tty

cat: write error: Bad file descriptor
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.stanwell.com/Trading/skeleton.git': No error
C:\>

Note: The credential manager works correctly with the full .exe installer

C:\> git clone https://github.stanwell.com/Trading/skeleton.git
Cloning into 'skeleton'...
Logon failed, use ctrl+c to cancel basic credential prompt.
Username for 'https://github.stanwell.com': dhirschfeld
Password for 'https://[email protected]':
remote: Counting objects: 28, done.
remote: Total 28 (delta 1), reused 1 (delta 1), pack-reused 26
Unpacking objects: 100% (28/28), done.

...however I'd like to user the BusyBox version as it fixes #1403/#1007 when running in a Windows Container.

What I'm hoping for is a git version which both allows cloning from a host-mapped volume and allows entering the username/password for the credential manager.

[dscho]

The full installer configures credential.helper = manager. What does this command tell you (with BusyBox-based MinGit)?

git config --show-origin credential.helper

I could imagine, though, that things would still fail with, say, multi-factor authorization, as I have no idea how the GitHub/Visual Studio Team Services/BitBucket/whatever helpers would behave if no graphical prompt was possible (IIRC most of them essentially show HTML pages): would they fall back to talking to the user via the console? Can they fall back that way?

The common strategy in Docker containers is to provide the credentials in different ways than via credential helpers: either use SSH keys, or configure the HTTPS remotes with credentials included. In your example, if the password was super-secret, then you would git clone https://dhirschfeld:[email protected]/Trading/skeleton.git.

(Of course, depending on your use case, you would want to ensure the password to be blotted out in whatever log your operations want to have.)

[dhirschfeld]

As shown in the working example at the bottom the credential manager just uses console input if you disallow the modal dialogue. This form of input is broken (bash tty error) though in the busybox version.

[dscho]

@dhirschfeld the Logon failed, use ctrl+c to cancel basic credential prompt message indeed comes from Git Credential Manager: https://github.com/Microsoft/Git-Credential-Manager-for-Windows/blob/3a85f87a872891dedf90d5fee211d102a63eb8ac/Cli-CredentialHelper/Program.cs#L329

The message error: failed to execute prompt script (exit code 1), in contrast, comes directly from git.exe:

git/compat/terminal.c

Line 141 in ef6d451

error("failed to execute prompt script (exit code %d)", code);

This suggests, very strongly even, that your BusyBox example does not even try to call the Git Credential Manager.

I suspected this. That is the entire reason I asked you to provide the information what git config --show-origin credential.helper prints.

I still would highly encourage you to actually work with me to resolve this. So far, I feel that half of what I said was simply ignored, and I doubt that this will lead to resolving the issue.

If my suspicion is correct, and that git config call reports an empty value for the credential.helper setting, the issue could be resolved by calling git config --system credential.helper manager in the BusyBox-based Git. And in that case, I would also get something out of this ticket, as I would have to dig further into the scripts generating the BusyBox-based Git so that the Git Credential Manager can be configured by default.

[dhirschfeld]

Sorry @dscho - I sent my initial response from my phone on the way into work with the full intention of following up on your other points. Other priorities took precedence however and this fell off my radar. I am keen to get this resolved though so am looking into it now...

Also, thanks for taking the time to look into my problem - it is much appreciated! :)

[dhirschfeld]

It does indeed seem that the credential helper isn't set:

C:\> git config --show-origin credential.helper
C:\>

If I set the credential.helper though I still get the same result :(

C:\> git config --global credential.helper manager

C:\> git clone https://github.stanwell.com/Trading/skeleton.git
Cloning into 'skeleton'...
Logon failed, use ctrl+c to cancel basic credential prompt.
bash: Unhandled device /dev/tty

cat: write error: Bad file descriptor
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.stanwell.com': No error

C:\> git config --show-origin credential.helper
file:C:/Users/ContainerAdministrator/.gitconfig manager

[dhirschfeld]

Note: The comment below from here seems a bit ominous:

BusyBox-w32 has no idea (yet) about Git Bash's interactive terminal, and therefore does not support any interactive usage

Happy to test anything else out in my Container environment if you have any suggestions...

[dhirschfeld]

The common strategy in Docker containers is to provide the credentials in different ways than via credential helpers: either use SSH keys, or configure the HTTPS remotes with credentials included

My usecase is probably a bit different than the usual - I'm trying to configure a JupyterLab interactive analytics environment which is deployed on Windows Containers. So in this case I want my users to be able to provide their own credentials using the (basic) credential manger:

I am aware that you can embed your username and password in the remote url but I'm trying to avoid having to do that if at all possible

[dscho]

I sent my initial response from my phone on the way into work with the full intention of following up on your other points. Other priorities took precedence however and this fell off my radar. I am keen to get this resolved though so am looking into it now...

Thank you for apologizing. I know how this feels, some days (in fact, most of them!) feel like drowning in an ocean, just trying to survive.

If I set the credential.helper though I still get the same result :(

That is a real bummer. Could you set GIT_TRACE=1 before running the clone operation? For me, the output looks like this (and I get a nice GitHub login window):

D:\test-git\MinGit-2.16.1.3-BusyBox-64>set GIT_TRACE=1

D:\test-git\MinGit-2.16.1.3-BusyBox-64>cmd\git clone https://github.com/dscho/none
12:21:20.549203 git.c:344               trace: built-in: git 'clone' 'https://github.com/dscho/none'
Cloning into 'none'...
12:21:20.755657 run-command.c:627       trace: run_command: 'remote-https' 'origin' 'https://github.com/dscho/none'
12:21:20.854183 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://github.com/dscho/none'
12:21:20.855184 run-command.c:627       trace: run_command: 'git-remote-https' 'origin' 'https://github.com/dscho/none'
12:21:22.780909 run-command.c:627       trace: run_command: 'git credential-manager get'
12:21:23.023968 git.c:576               trace: exec: 'git-credential-manager' 'get'
12:21:23.024970 run-command.c:627       trace: run_command: 'git-credential-manager' 'get'

Also, I wonder whether you see different behavior between running these commands in Jupyter as opposed to a CMD window?

[dhirschfeld]

The output with GIT_TRACE=1 is below:

C:\> $env:GIT_TRACE=1
C:\> git clone https://github.stanwell.com/Trading/skeleton.git
15:44:15.586700 git.c:344               trace: built-in: git 'clone' 'https://github.stanwell.com/Trading/skeleton.git'
Cloning into 'skeleton'...
15:44:15.619704 run-command.c:627       trace: run_command: 'remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
15:44:15.644699 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
15:44:15.645699 run-command.c:627       trace: run_command: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
15:44:15.819711 run-command.c:627       trace: run_command: 'git credential-manager get'
15:44:16.093720 git.c:576               trace: exec: 'git-credential-manager' 'get'
15:44:16.094720 run-command.c:627       trace: run_command: 'git-credential-manager' 'get'
Logon failed, use ctrl+c to cancel basic credential prompt.
15:44:17.593774 run-command.c:627       trace: run_command: 'bash' '-c' 'cat >/dev/tty && read -r line </dev/tty && echo "$line"'
15:44:17.596773 run-command.c:627       trace: run_command: 'C:\Miniconda3\Library\mingw64\bin\busybox.exe' '--help'
bash: Unhandled device /dev/tty

cat: write error: Bad file descriptor
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.stanwell.com': No error

The bash command is a pretty suspicious since bash (or WSL) isn't installed on the server.

[dhirschfeld]

I get the same results if I try directly from powershell in the container rather than through the JupyterLab/xterm.js terminal:

C:\> $env:USERNAME
ContainerAdministrator
C:\> git config --global credential.modalPrompt never
C:\> git config --global credential.helper manager
C:\> git config --show-origin credential.helper
file:C:/Users/ContainerAdministrator/.gitconfig manager
C:\> $env:GCM_INTERACTIVE='NEVER'
C:\> $env:GCM_PRESERVE_CREDS='TRUE'
C:\> $env:GIT_TRACE=1
C:\> $env:GCM_TRACE=1
C:\> git clone https://github.stanwell.com/Trading/skeleton.git
16:09:29.845751 git.c:344               trace: built-in: git 'clone' 'https://github.stanwell.com/Trading/skeleton.git'
Cloning into 'skeleton'...
16:09:29.892753 run-command.c:627       trace: run_command: 'remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
16:09:29.916748 git.c:576               trace: exec: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
16:09:29.917748 run-command.c:627       trace: run_command: 'git-remote-https' 'origin' 'https://github.stanwell.com/Trading/skeleton.git'
16:09:30.120757 run-command.c:627       trace: run_command: 'git credential-manager get'
16:09:30.238414 git.c:576               trace: exec: 'git-credential-manager' 'get'
16:09:30.239413 run-command.c:627       trace: run_command: 'git-credential-manager' 'get'
16:09:30.550416 ...\Common.cs:527       trace: [Main] git-credential-manager (v1.14.0) 'get'
16:09:30.761424 ...\Where.cs:240        trace: [FindGitInstallations] found 0 Git installation(s).
16:09:31.085437 ...Configuration.cs:405 trace: [LoadGitConfiguration] git All config read, 4 entries.
16:09:31.094443 ...\Where.cs:240        trace: [FindGitInstallations] found 0 Git installation(s).
16:09:31.097441 ...Configuration.cs:405 trace: [LoadGitConfiguration] git All config read, 4 entries.
16:09:31.122437 ...\Common.cs:395       trace: [LoadOperationArguments] GCM_INTERACTIVE = 'NEVER'.
16:09:31.123442 ...\Common.cs:437       trace: [LoadOperationArguments] preserve = 'True'.
16:09:31.678467 ...\Common.cs:77        trace: [CreateAuthentication] detecting authority type for 'https://github.stanwell.com/'.
16:09:31.696466 ...uthentication.cs:137 trace: [GetAuthentication] not github.com, authentication creation aborted.
16:09:31.698469 ...\Common.cs:172       trace: [CreateAuthentication] authority for 'https://github.stanwell.com/' is basic with NTLM=Auto.
16:09:31.721468 ...\Common.cs:565       trace: [QueryCredentials] credentials for 'https://github.stanwell.com/' not found.
16:09:31.722493 ...\Common.cs:495       trace: [LogEvent] Failed to retrieve credentials for 'https://github.stanwell.com/'.
16:09:31.767463 ...\Program.cs:265      trace: [Get] Logon failed, use ctrl+c to cancel basic credential prompt.
Logon failed, use ctrl+c to cancel basic credential prompt.
16:09:31.806468 run-command.c:627       trace: run_command: 'bash' '-c' 'cat >/dev/tty && read -r line </dev/tty && echo "$line"'
16:09:31.810471 run-command.c:627       trace: run_command: 'C:\Miniconda3\Library\mingw64\bin\busybox.exe' '--help'
bash: Unhandled device /dev/tty

cat: write error: Bad file descriptor
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.stanwell.com': No error
C:\>

[dhirschfeld]

That is a real bummer.

Yeah, it would have been great if it was as simple as setting the credential manager! 😄

I still hold out a glimmer of hope that something in the trace may jump out as an obvious/easy fix but it doesn't seem too likely 😞

[dscho]

C:> $env:GCM_INTERACTIVE='NEVER'

Is this intentional? I would have expected this to tell the Git Credential Manager that it should not ask for the credentials...

[dscho]

The bash command is a pretty suspicious since bash (or WSL) isn't installed on the server.

It is included in MinGit (although not in BusyBox-based MinGit). And Git for Windows does not yet have code to special-case the BusyBox case and try a different tack on accessing the terminal in that case.

[dhirschfeld]

$env:GCM_INTERACTIVE='NEVER' appears to be necessary to force the git credential manager to use basic input i.e. to prompt for username and password at the console. Without that it just hangs, I believe because it tries to open a GUI dialog when there is no GUI on the containers.

Using the same configuration does work (prompts for username & password) with the full/normal git installer but then you run into #1403 so it's a case of pick your poison.

Git for Windows does not yet have code to special-case the BusyBox case and try a different tack on accessing the terminal in that case

I suspected that may be the case. I guess this issue can be left open as a reminder that it would be a useful feature to add to the BusyBox git.

[dscho]

$env:GCM_INTERACTIVE='NEVER' appears to be necessary to force the git credential manager to use basic input i.e. to prompt for username and password at the console. Without that it just hangs, I believe because it tries to open a GUI dialog when there is no GUI on the containers.

Using the same configuration does work (prompts for username & password) with the full/normal git installer but then you run into #1403 so it's a case of pick your poison.

Hmm, okay. My mistake, sorry for that.

Git for Windows does not yet have code to special-case the BusyBox case and try a different tack on accessing the terminal in that case

I suspected that may be the case. I guess this issue can be left open as a reminder that it would be a useful feature to add to the BusyBox git.

I'd rather fix the main problem first: ensure that the Win32 Console can be accessed by GCM even when it is called via BusyBox-based MinGit.

Let me try to set up something (will take a while, that's why I tried to avoid that so far, also because I am not altogether sure that I can reproduce the issue).

[dscho]

Let me try to set up something (will take a while, that's why I tried to avoid that so far, also because I am not altogether sure that I can reproduce the issue).

I can see the same issue you describe in a Powershell without Docker containers. Now, how do I debug this ;-)

[dscho]

Oy vey, I have been chasing the wrong goose all the time! This is what I get with non-BusyBox-backed MinGit:

[...]
15:28:28.539347 ...\Common.cs:142       trace: [CreateAuthentication] authority for 'https://github.com/' is GitHub.
15:28:28.566310 ...\Console.cs:47       trace: [CredentialPrompt] not a tty detected, abandoning prompt.
15:28:28.566817 ...uthentication.cs:224 trace: [InteractiveLogon] interactive logon for 'https://github.com/' failed.
15:28:28.567309 ...\Common.cs:666       trace: [QueryCredentials] credentials for 'https://github.com/' not found.
15:28:28.575862 ...\Common.cs:495       trace: [LogEvent] Failed to retrieve GitHub credentials for 'https://github.com/'.
15:28:28.590657 ...\Program.cs:265      trace: [Get] Logon failed, use ctrl+c to cancel basic credential prompt.
Logon failed, use ctrl+c to cancel basic credential prompt.
15:28:28.627658 run-command.c:627       trace: run_command: 'bash' '-c' 'cat >/dev/tty && read -r line </dev/tty && echo "$line"'
Username for 'https://github.com':

Do you notice how it says [CredentialPrompt] not a tty detected, abandoning prompt and falls back to Git's own prompt, too? So the problem is not so much that Git Credential Manager fails to retrieve credentials from via terminal in BusyBox-based MinGit. It is more that it fails to retrieve credentials from terminal when Git was called from Powershell, whether BusyBox is involved or MinGit does not matter much. And the fall-back with Bash simply does not work with BusyBox, is all.

FWIW I found another environment variable you definitely want to set in your scenario:

$env:GCM_MODAL_PROMPT='No'

[dscho]

Good news! In a joint debugging session with @whoisj, we determined the root cause in Git Credential Manager. @dhirschfeld are you willing to test an experimental build (which I should have ready either later today, but definitely before noon tomorrow UTC+0100)?

Most likely I will also have a fix for the fallback code in Git at the same time.

[dscho]

credential-helper-w-tty-fix.zip

Okay, so this is the fixed Credential Manager. If you unpack this into your BusyBox-based MinGit's mingw64\libexec\git-core directory (overwriting existing files), it should start working.

[dhirschfeld]

Thanks for getting to the bottom of this @dscho! ...despite my efforts to confuse the issue with docker and busybox! 😜

I'll certainly test that out on my setup. May not get to it today but if not, will test it over the weekend...

[dhirschfeld]

FWIW I found another environment variable you definitely want to set in your scenario

I was instead using git config --global credential.modalPrompt never. That was the first thing I tried and was a little surprised that it wasn't sufficient to stop the dialog popping up. That's when I found out about the other environment variables which seemed to do the trick:

C:\> $env:GCM_INTERACTIVE='NEVER'
C:\> $env:GCM_PRESERVE_CREDS='TRUE'

If there's an equivalent git config for those I'd probably use that instead but I don't know that there is? I'll look into that, but it's not a problem if I do have to specify env vars to get the behaviour I'm after...

[dscho]

(This Git Credential Manager version was built from microsoft/Git-Credential-Manager-for-Windows#554.)

[dscho]

FWIW I found another environment variable you definitely want to set in your scenario

I was instead using git config --global credential.modalPrompt never. That was the first thing I tried and was a little surprised that it wasn't sufficient to stop the dialog popping up

I was surprised, too, that's why I studied the source code of Git Credential Manager and found GCM_MODAL_PROMPT.

[whoisj]

was instead using git config --global credential.modalPrompt never. That was the first thing I tried and was a little surprised that it wasn't sufficient to stop the dialog popping up.

That is likely because the only legal values are "true" or "false". 😏

Configuration Setting Names: ModalPrompt

[whoisj]

(This Git Credential Manager version was built from microsoft/Git-Credential-Manager-for-Windows#554.)

Oh and for those wondering, this fix will be part of GCM v1.15.0 (releases sometime in March).

Thanks @dscho for the help, and thanks to @dhirschfeld for the inspiration to root cause and resolve the issue.