git-bash "basic credential prompt" will encode passwords in non-UTF8

[celalsahin]

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options
git version 2.20.1.windows.1
cpu: x86_64
built from commit: 7c9fbc0
sizeof-long: 4
sizeof-size_t: 8

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?
  Win 10 64-bit
  C:\Users\tr1s4323>ver

Microsoft Windows [Version 10.0.15063]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
C:\Users\tr1s4323>type "C:\Program Files\Git\etc\install-options.txt"
Editor Option: VIM
Custom Editor Path:
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled

 ### Details

 - Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Bash


$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=

 - What commands did you run to trigger this issue? If you can provide a
   [Minimal, Complete, and Verifiable example](http://stackoverflow.com/help/mcve)
   this will help us understand the issue.

Short: If I don't use "windows credential helper" (i.e. credential.helper manager) for the password, 
special characters (e.g. § - õ) does not work in password. 

Long: "OpenSSH" prompt(pop-up) asking for the password does not encode the password in UTF8 
although all is set to use UTF8 in shell. I just recognized this via "gitlab" installation. This can be 
reproduced, e.g.: (just enter "§" for the username & password in basic cred. prompt.)

$ git clone https://gitlab.com/celal.sahin/celal_test_123.git
Cloning into 'celal_test_123'...
Logon failed, use ctrl+c to cancel basic credential prompt.
Username for 'https://gitlab.com': §
fatal: unable to access 'https://gitlab.com/celal.sahin/celal_test_123.git/': The requested URL returned error: 500
--

--> I had credentials manager set so I canceled it to get the "basic credential prompt", which is 
"OpenSSH" window in my case. This will trigger the error in gitlab logs as:

ActiveRecord::StatementInvalid (PG::CharacterNotInRepertoire: ERROR:  invalid byte sequence for 
encoding "UTF8": 0xa7

Using "Windows Credential Manager" passwords with "§" "õ" etc. would succeed. 

 - What did you expect to occur after running these commands?

I expect both "basic cred prompt" & "Windows Cred. Manager" act the same way & follow the encoding
 settings of the terminal/system.

 - What actually happened instead?

"Basic Cred Prompt" does not follow the system encoding settings.

[celalsahin]

Any chance this gets any attention?

[dscho]

@celalsahin does it work okay if you do this in Git CMD?

[celalsahin]

Nope again it throws "500" (not 401 - which is access denied) In logs:

PG::Error (incomplete multibyte character
):

Again an error about charset - not the same error as git-bash

btw when I open "git cmd" it says "deprecated"

[dscho]

btw when I open "git cmd" it says "deprecated"

Yes. We could probably "de-deprecate" it by now because a contributor found a way to fix the issue that I previously thought was unfixable: Git CMD nowadays does not pick up a git.ps1 in the current directory when calling, say, git show.

Nope again it throws "500" (not 401 - which is access denied) In logs:
[...]

I think we need a better reproducer, to allow for efficient debugging (keep in mind that there are plenty of components involved, and the culprit could be in any of them, even in a subset of them).

Unless I am mistaken, the function that reads the user input is in git_terminal_prompt(). The only caller of that function is git_prompt(). And there are only two callers of that: https://github.com/git-for-windows/git/blob/v2.25.0.windows.1/builtin/bisect--helper.c#L331 and https://github.com/git-for-windows/git/blob/v2.25.0.windows.1/credential.c#L128. The former does not show the user input, so we cannot use it. The latter call site is inside the credential_ask_one() function which in turn is called by the credential_getpass() which in turn is called by the credential_fill() function.

This is where it gets interesting: the credential_fill() function is called by several callers, and the most interesting for our "let me reproduce this quick" purposes is most likely this one: https://github.com/git-for-windows/git/blob/v2.25.0.windows.1/builtin/credential.c#L24

The way to trigger this is to call git credential fill, but that requires some input on stdin:

$ echo host=http://abc.com | git -c credential.helper= credential fill

However, when I call that, I immediately get this popup-window:

Reading back in this ticket, I realize that is exactly what you talked about! I thought you were talking about the text-mode input, but instead you are talking about git gui--askpass (which is written in Tcl/Tk). And I think that the reason this happens is exactly this: it is a Tcl/Tk user dialog, and I am not sure that it knows that we are interested in UTF-8-encoded input.

So a much quicker reproducer is to call git gui--askpass and type in e.g. Hello äöü world. You will immediately see that the output Hello world is missing the umlauts.

This was the easy part of the investigation, and unfortunately, I spent all the time I can afford on this ticket for the foreseeable future on the analysis so far 😦

[dscho]

Oh, right, I should mention that the source code for git gui--askpass is here: https://github.com/git-for-windows/git/blob/v2.25.0.windows.1/git-gui/git-gui--askpass

[lbonanomi]

Please forgive a naive question, but isn't this just a case of coercing the $::answer variable to UTF-8?

Adding the line:

set ::answer [encoding convertto utf-8 $::answer]

Lets git-gui--askpass pass-through the string "¡Hëlló Wõrld!" letter-perfect.

This may not be the right place for it, but I opened a PR for the same. @lbonanomi is a forgetful sort.