[BUG]: "could not read Password" error in `checkout` task when `fetchFilter` is provided

[afscrome]

What happened?

If you add a fetchFilter (from #4693) to the checkout task (as well as enabling the AGENT_USE_FETCH_FILTER_IN_CHECKOUT_TASK knob), then the checkout manages to fetch objects from the remotes, but fails on the final git checkout task with an authentication issue:

There is a warning in the result suggesting changing fetchDepth, however I get the same issue regardless of whether I set fetch depth

Repo pipeline:

job: demo
displayName: 🏗️ Build
steps:
  - checkout: self
    fetchFilter: tree:0
    #fetchDepth: 0

Versions

Current agent version: '3.241.0'

Operating System
Ubuntu
22.04.4
LTS

Runner Image
Image: ubuntu-22.04
Version: 20240616.1.0
Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240616.1/images/ubuntu/Ubuntu2204-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240616.1

Runner Image Provisioner
2.0.370.1
Current image version: '20240616.1.0'

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu 22.04.04 LTS

Version controll system

Git

Relevant log output

Starting: Checkout REPO@main to s
==============================================================================
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Syncing repository: REPO (Git)
git version
git version 2.45.2
git lfs version
git-lfs/3.5.1 (GitHub; linux amd64; go 1.21.8)
git init "/home/vsts/work/1/s"
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
Initialized empty Git repository in /home/vsts/work/1/s/.git/
git remote add origin https://ORG@dev.azure.com/ORG/PROJECT/_git/REPO
git config gc.auto 0
git config core.longpaths true
git config --get-all http.https://ORG@dev.azure.com/ORG/PROJECT/_git/REPO.extraheader
git config --get-all http.extraheader
git config --get-regexp .*extraheader
git config --get-all http.proxy
git config http.version HTTP/1.1
git --config-env=http.extraheader=env_var_http.extraheader fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin --depth=1 --filter=tree:0 +edb49c62b56ceb393542ebb413c58b39130fd07e:refs/remotes/origin/edb49c62b56ceb393542ebb413c58b39130fd07e
remote: Azure Repos        
remote: 
remote: Found 1 objects to send. (10 ms)        
Receiving objects: 100% (1/1)
Receiving objects: 100% (1/1), 184 bytes | 184.00 KiB/s, done.
From https://dev.azure.com/ORG/PROJECT/_git/REPO
 * [new ref]         edb49c62b56ceb393542ebb413c58b39130fd07e -> origin/edb49c62b56ceb393542ebb413c58b39130fd07e
git --config-env=http.extraheader=env_var_http.extraheader fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin --depth=1 --filter=tree:0 +edb49c62b56ceb393542ebb413c58b39130fd07e
remote: Azure Repos        
remote: 
remote: Found 0 objects to send. (19 ms)        
From https://dev.azure.com/ORG/PROJECT/_git/REPO
 * branch            edb49c62b56ceb393542ebb413c58b39130fd07e -> FETCH_HEAD
git checkout --progress --force refs/remotes/origin/edb49c62b56ceb393542ebb413c58b39130fd07e
fatal: could not read Password for 'https://ORG@dev.azure.com': terminal prompts disabled
fatal: could not fetch 7f12c57ea65125ee8da773d3de56ea12bc338daa from promisor remote
##[warning]Git checkout failed on shallow repository, this might because of git fetch with depth '1' doesn't include the checkout commit 'refs/remotes/origin/edb49c62b56ceb393542ebb413c58b39130fd07e'. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=829603)
##[error]Git checkout failed with exit code: 128
Finishing: Checkout REPO@main to s

[afscrome]

Setting AGENT_USE_FETCH_FILTER_IN_CHECKOUT_TASK no longer seems to be required, but the issue still occurs.

I believe the problem is the final checkout call is missing credentials (presumably --config-env=http.extraheader=env_var_http.extraheader)

git checkout --progress --force refs/remotes/origin/edb49c62b56ceb393542ebb413c58b39130fd07e

If you add a second checkout step before with persistCredential: true, then the treeless clone works correctly.

  steps:
    # Fake checkout in order to get credentials persisted, after which point the second treeless clone works
    - checkout: self
      persistCredentials: true
      path: fake

    - checkout: self
      fetchFilter: tree:0
      fetchDepth: 1
      fetchTags: false

[ivanduplenskikh]

Hi @afscrome, thanks for reporting.
I have reproduced the issue.
It arises when the checkout command is executed on an agent that either lacks access to the credential-manager or does not have the necessary credentials for the requested repository in credential-manager.
I will keep you updated as I make further progress.

[ivanduplenskikh]

@afscrome, I've linked the PR that addresses the issue and is currently under review.

[ivanduplenskikh]

@afscrome, could you kindly verify if the issue with your pipeline has been addressed?

[richshadman]

We are still seeing this issue @afscrome.

We are on agent version 3.246.0 on a private hosted agent. Our situation is a little bit complex - we are doing comparisons between source and target commits during a PR to determine impacted projects for testing purposes.

To do this we first check out the target branch, get the commit id, then the source branch, do a comparison between them and get a file list to check against our projects. We have a very large history and therefore are taking advantage of the support for fetch filtering using blob:none to avoid the lengthy download.

When we get to checking out the source commit, if there are blobs that are not already present in the local repository, there is a fetch to the promisor remote (origin) that i believe is still missing the credentials. Please see the screenshot below:

This doesn't happen all the time, only when there is gap that has to be fetched. In these cases having the developer rebase onto the target usually resolves the issue (avoiding the promisor remote fetch). We have also added powershell script to perform some of the git commands directly which seems to work with the persisted credentials just fine.

EDIT for clarity in the picture:

I know I blacked out most of the commit id's however I wanted to point out the error happens on a different id (4c5) entirely than the id getting checked out (ee0). The issue is not the initial fetch call but the additional fetches I believe it triggers for the missing blob history (which is based on my rudimentary understanding of the whole partial clone fetch workflow - essentially fetched on demand). The error ultimately reported erroneously points to the target of the checkout (ee0) and not the id that actually causes the failure.

[ivanduplenskikh]

@richshadman, could you please provide a reproducible sample of the pipeline?

[ivanduplenskikh]

@afscrome, I'm closing this issue for the time being. Feel free to re-open this issue if you have any additional questions.

@richshadman, could you please clarify if you are using the checkout task or if you are executing the checkout by custom script?

[asyncmeow]

Hello!
I'm having a similar issue with a fairly minimal pipeline - just a few steps split over two jobs and two stages (one job per stage):

# .azure-pipelines.yml
trigger:
  - '*'

pool:
  name: # [azure devops managed pool name]

stages:
  - stage: DatadogMetadata
    displayName: "Upload Git metadata to Datadog"
    jobs:
      - job: DatadogMetadata
        displayName: "Upload Git metadata to Datadog"
        steps:
          - task: Bash@3
            displayName: "Upload metadata"
            env:
              DATADOG_API_KEY: $(ddApiKey)
            inputs:
              targetType: 'inline'
              script: |
                curl -L --fail "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_linux-x64" --output "/usr/local/bin/datadog-ci"
                chmod +x /usr/local/bin/datadog-ci
                /usr/local/bin/datadog-ci git-metadata upload

  - template: ./azure-pipelines-ci/service-deploy.yml
    parameters:
      environment: staging
      api: customer
---
# ./azure-pipelines-ci/service-deploy.yml
parameters:
  environment: ''
  api: ''

stages:
  - stage: Deploy_${{ parameters.environment }}_${{ parameters.api }}
    displayName: "Deploy - ${{ parameters.environment }} ${{ parameters.api }}"
    jobs:
      - job: Deploy
        displayName: "Deploy"
        steps:
          - task: Docker@2
            displayName: 'Login to Container Registry'
            inputs:
              containerRegistry: # container registry name here
              command: 'login'
          - task: DotNetCoreCLI@2
            inputs:
              command: 'publish'
              publishWebProjects: false
              arguments: '--os linux --arch x64 /t:PublishContainer ./APIs/${{ parameters.api }}Api/${{ parameters.api }}Api.csproj'
              zipAfterPublish: false
              modifyOutputPath: false

yet, when Azure DevOps tries to checkout the repository:

git --config-env=http.extraheader=env_var_http.extraheader fetch --force --tags --prune --prune-tags --progress --no-recurse-submodules origin   +cb3c27436193b148fd065305592de49f455fbd40
From https://dev.azure.com/[OrganizationName]/[ProjectName]/_git/[RepoName]
 * branch            cb3c27436193b148fd065305592de49f455fbd40 -> FETCH_HEAD
git checkout --progress --force cb3c27436193b148fd065305592de49f455fbd40
fatal: could not read Password for 'https://[OrganizationName]@dev.azure.com': terminal prompts disabled
fatal: could not fetch a8cde9df9bf2b3de9d35a24bb36518fa58b0651d from promisor remote
##[error]Git checkout failed with exit code: 128
Finishing: Checkout [RepoName]@main to s

This is with shallow fetch disabled as well, which I tried after getting this error:

##[warning]Git checkout failed on shallow repository, this might because of git fetch with depth '1' doesn't include the checkout commit 'refs/remotes/origin/cb3c27436193b148fd065305592de49f455fbd40'. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=829603)

Any ideas how I could fix this?