yarn 3 ignores npmAuthTokens in ~/.yarnrc.yml: "Invalid authentication (as an anonymous user)" when using private repo, despite `yarn npm login`

[gustafc]

TL;DR: yarn 3.2.0 seems to ignore auth tokens in $HOME/.yarnrc.yml, since the token generated by yarn npm login has to be manually added to $PROJECT/.yarnrc.yml to authenticate when doing yarn install. There's a reproduction script at the end!

I'm trying to upgrade a project from yarn 1.x to modern yarn using the migration checklist, and I'm running into trouble at the step where I run yarn install:

1. Run npm install -g yarn to update the global yarn version to latest v1
2. Go into your project directory
3. Run yarn set version berry to enable v2 (cf Install for more details)
4. If you used .npmrc or .yarnrc, you'll need to turn them into the new format (see also 1, 2)
5. Add nodeLinker: node-modules in your .yarnrc.yml file
6. Commit the changes so far (yarn-X.Y.Z.js, .yarnrc.yml, ...)
7. Run yarn install to migrate the lockfile

We install all modules from a private Nexus repository, which houses our private packages while also acting as a proxy to the public NPM repo. This Nexus repo requires authentication at all times, which is where the troubles begin.

This is the project's .yarnrc.yml:

yarnPath: .yarn/releases/yarn-3.2.0.cjs
nodeLinker: node-modules
npmRegistryServer: "https://nexus.example.com/repository/npm-all"
npmAlwaysAuth: true
npmRegistries:
  "https://nexus.example.com/repository/npm-all":
    npmAlwaysAuth: true

If I try to install without logging in, it predictably fails since I haven't logged in to the Nexus server:

$ yarn install
➤ YN0000: ┌ Resolution step
➤ YN0041: │ @testing-library/jest-dom@npm:5.16.4: Invalid authentication (as an anonymous user)
➤ YN0000: └ Completed in 0s 513ms
➤ YN0000: Failed with errors in 0s 521ms

So I login:

$ yarn npm login
➤ YN0000: Logging in to https://nexus.example.com/repository/npm-all

✔ Username: · my-name
✔ Password: · ****************

➤ YN0000: Successfully logged in
➤ YN0000: Done in 17s 175ms

Seems to have gone well, and it has generated a $HOME/.yarnrc.yml:

npmRegistries:
  "https://nexus.example.com/repository/npm-all":
    npmAuthToken: NpmToken.00000000-1111-2222-3333-444444444444

But yarn install still fails, and I actually don't seem to be logged in:

$ yarn install
➤ YN0000: ┌ Resolution step
➤ YN0041: │ @testing-library/jest-dom@npm:5.16.4: Invalid authentication (as an anonymous user)
➤ YN0000: └ Completed in 0s 523ms
➤ YN0000: Failed with errors in 0s 532ms

$ yarn npm whoami
➤ YN0033: No authentication configured for request
➤ YN0000: Failed with errors in 0s 4ms

The only workaround I've found is copying the npmAuthToken from $HOME/.yarnrc.yml to $PROJECT/.yarnrc.yml:

$ cat $PROJECT/.yarnrc.yml
yarnPath: .yarn/releases/yarn-3.2.0.cjs
nodeLinker: node-modules
npmRegistryServer: "https://nexus.example.com/repository/npm-all"
npmAlwaysAuth: true
npmRegistries:
  "https://nexus.example.com/repository/npm-all":
    npmAuthToken: NpmToken.00000000-1111-2222-3333-444444444444
    npmAlwaysAuth: true

$ yarn npm whoami
➤ YN0000: my-name
➤ YN0000: Done in 0s 74ms

This is obviously not something I want to do. How can I tell yarn to use the auth token that yarn npm login creates?

Versions used:

$ yarn -v
3.2.0
$ node -v
v14.16.0

EDIT: So I managed to set up a script that reproduces this issue. It starts a new Nexus server through docker, sets up yarn project, logs in, and tries to add a package.

#!/bin/bash
set -exuo pipefail

UNIQUE_STRING=$(date '+%Y%m%d-%H%M%S')
PROJECT=/tmp/yarn3-auth-repro-$UNIQUE_STRING
# Step 1: Create a new project in /tmp
mkdir -p $PROJECT
cd $_
yarn init -2
echo '
unsafeHttpWhitelist:
  - localhost
npmRegistryServer: "http://localhost:8081/repository/npm-all"
npmAlwaysAuth: true
' >>.yarnrc.yml

# Step 2: Set up Nexus
NEXUS_CONTAINER=nexus-repro-container-$UNIQUE_STRING

docker run --rm --detach -p 8081:8081 --name $NEXUS_CONTAINER sonatype/nexus3@sha256:66fe12b1eb3e97bae72eb3c2c4e436499d41ff144cdfd1dcd0718df738304732
function cleanup {
  read -p "Stop Nexus docker container $NEXUS_CONTAINER? (y/n) " STOP_CONTAINER
  if [[ $STOP_CONTAINER == y* ]]; then
    docker stop $NEXUS_CONTAINER
  else
    echo "You can play around in the project by going to the following directory:"
    echo "  cd $PROJECT"
    echo "To stop the Nexus container, run:"
    echo "  docker stop $NEXUS_CONTAINER"
  fi
}
trap cleanup EXIT

while ! docker logs $NEXUS_CONTAINER | grep 'Started Sonatype Nexus OSS' ; do echo "Waiting for $NEXUS_CONTAINER to start..."; sleep 2; done
printf '\a'
NEXUS_USER=admin
NEXUS_PASS="$(docker exec $NEXUS_CONTAINER cat /nexus-data/admin.password)"
# # Change admin password
# curl --fail --verbose -u "$NEXUS_USER:$NEXUS_PASS" 'http://localhost:8081/service/rest/internal/ui/onboarding/change-admin-password' -X PUT --data-raw 'admin123'
# sleep 1
# NEXUS_PASS=admin123
# Disallow anonymous access
curl --fail --verbose -u "$NEXUS_USER:$NEXUS_PASS" 'http://localhost:8081/service/extdirect' -X POST -H 'Content-Type: application/json' --data-raw '{"action":"coreui_AnonymousSettings","method":"update","data":[{"enabled":false,"userId":"anonymous","realmName":"NexusAuthorizingRealm"}],"type":"rpc","tid":14}'

# Create npm repo proxy
curl --fail --verbose -u "$NEXUS_USER:$NEXUS_PASS" 'http://localhost:8081/service/extdirect' -X POST -H 'Content-Type: application/json' --data-raw '{"action":"coreui_Repository","method":"create","data":[{"attributes":{"npm":{"removeNonCataloged":false,"removeQuarantinedVersions":false},"proxy":{"remoteUrl":"https://registry.npmjs.org","contentMaxAge":1440,"metadataMaxAge":1440},"httpclient":{"blocked":false,"autoBlock":true,"connection":{"useTrustStore":false}},"storage":{"blobStoreName":"default","strictContentTypeValidation":true},"negativeCache":{"enabled":true,"timeToLive":1440},"cleanup":{"policyName":[]}},"name":"npm-proxy","format":"","type":"","url":"","online":true,"routingRuleId":"","authEnabled":false,"httpRequestSettings":false,"recipe":"npm-proxy"}],"type":"rpc","tid":31}'

# Create npm repo group (which is my real setup)
curl --fail --verbose -u "$NEXUS_USER:$NEXUS_PASS" 'http://localhost:8081/service/extdirect' -X POST -H 'Content-Type: application/json' --data-raw '{"action":"coreui_Repository","method":"create","data":[{"attributes":{"storage":{"blobStoreName":"default","strictContentTypeValidation":true},"group":{"memberNames":["npm-proxy"]}},"name":"npm-all","format":"","type":"","url":"","online":true,"recipe":"npm-group"}],"type":"rpc","tid":44}'

# Make NPM login work <https://issues.sonatype.org/browse/NEXUS-20170>
curl --fail --verbose -u "$NEXUS_USER:$NEXUS_PASS" 'http://localhost:8081/service/extdirect' -X POST -H 'Content-Type: application/json' --data-raw '{"action":"coreui_RealmSettings","method":"update","data":[{"realms":["NexusAuthenticatingRealm","NexusAuthorizingRealm","NpmToken"]}],"type":"rpc","tid":43}'


# Step 3: Login
(echo "$NEXUS_USER"; sleep 2; echo "$NEXUS_PASS") | yarn npm login
yarn npm whoami

# Step 4: Add any random package to make sure auth works (this will fail)
yarn add mkdirp@latest

If you run this script and don't shut down Nexus at the end (you will be asked if you want to), you can cd to the directory the script created, and manually add the token from $HOME/.yarnrc.yml to $PROJECT/.yarnrc.yml, like so:

unsafeHttpWhitelist:
  - localhost
npmRegistryServer: "http://localhost:8081/repository/npm-all"
npmAlwaysAuth: true
npmRegistries:
  "http://localhost:8081/repository/npm-all":
    npmAlwaysAuth: true
    npmAuthToken: NpmToken.xxxxxxxxxxxxx

[gustafc]

Given the lack of reactions to this issue, I've added a Bash script to the original post which reproduces the error, using a local Nexus running in Docker. Enjoy!

[HeatherFlux]

I'm feeling your pain on this one as well. It seems to lose its auth credentials when dealing with non root packages in my monorepo.

[rosskevin]

I am transitioning our projects to v3, and I have found the failure in authentication to the github package repo.

nodeLinker: pnp

npmScopes:
  alienfast:
    npmAlwaysAuth: true
    npmRegistryServer: "https://npm.pkg.github.com"

yarnPath: .yarn/releases/yarn-3.2.0.cjs

~/p/hello-world-yarn ❯❯❯ yarn npm publish
➤ YN0000: .editorconfig
➤ YN0000: Readme.md
➤ YN0000: bin.js
➤ YN0000: example.gif
➤ YN0000: lib/index.js
➤ YN0000: package.json
➤ YN0000: Package archive published
➤ YN0000: Done in 2s 773ms
~/p/hello-world-yarn ❯❯❯ npm whoami
rosskevin
~/p/hello-world-yarn ❯❯❯ yarn npm whoami
➤ YN0033: No authentication configured for request
➤ YN0000: Failed with errors in 0s 4ms

As you can see, I can publish a private package, but yarn npm whoami reports no auth whereas npm whoami reports my rosskevin user.

I investigated this path because previous packages published on https://npm.pkg.github.com/ are reporting unavailable with misleading messages such as:

~/p/hello-world-yarn ❯❯❯ yarn add @alienfast/eslint-config
➤ YN0000: ┌ Resolution step
➤ YN0033: │ eslint-config-prettier@npm:^8.5.0: No authentication configured for request
➤ YN0000: └ Completed in 3s 875ms
➤ YN0000: Failed with errors in 3s 880ms

whereas npm works fine:

~/p/hello-world-yarn ❯❯❯ npm add @alienfast/eslint-config                                                                                                                                                            ✘ 1 

added 195 packages, and audited 196 packages in 13s

64 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

TL;DR yarn 3 authentication is not working with older github artifacts, and yarn npm whoami is misreporting No authentication configured for request even though yarn npm publish successfully publishes to the github repository!

~/p/hello-world-yarn ❯❯❯ yarn npm whoami --scope alienfast                                                                                                                                                           ✘ 1 
➤ YN0000: rosskevin
➤ YN0000: Done in 0s 348ms

(as an aside, I tried publishing a current @alienfast/eslint-config with yarn npm publish which is also successful but no yarn command can pull from that repo, even though npm can).

SO ... WHY can I publish but not install from the same config?

[rosskevin]

Bizarrely, yarn config shows opposite of what I have scoped:

➤ YN0000: npmRegistries                 Map(1) { 'https://npm.pkg.github.com' => Map(3) { 'npmAlwaysAuth' => false, 'npmAuthIdent' => null, 'npmAuthToken' => '********' } }
➤ YN0000: npmRegistryServer             'https://registry.yarnpkg.com'
➤ YN0000: npmScopes                     Map(1) { 'alienfast' => Map(6) { 'npmAlwaysAuth' => true, 'npmAuthIdent' => null, 'npmAuthToken' => null, 'npmAuditRegistry' => null, 'npmPublishRegistry' => null, 'npmRegistryServer' => 'https://npm.pkg.github.com' } }
➤

npmAlwaysAuth is clearly set to true under the alienfast scope IN the project's yarnrc.yml.

[rosskevin]

Ok, so this seems to look like something attempted to be solved by the deep merge PR from #2106 which was rejected due to major breaking changes?

This is a bug, confirmed. The only way I can (so far) get yarn config to appear properly and yarn add to work properly is to include my token in the project's yarnrc.yml (huge security hole) and specify my auth with BOTH the npmRegistries key AND npmScopes.

e.g.

nodeLinker: pnp

npmRegistries:
  "https://npm.pkg.github.com":
    npmAuthToken: xxxxx
    npmAlwaysAuth: true

npmScopes:
  alienfast:
    npmAlwaysAuth: true
    npmRegistryServer: "https://npm.pkg.github.com"

yarnPath: .yarn/releases/yarn-3.2.0.cjs

^^^ this works as a project yarnrc.yml but is unusable from a security perspective.

[gustafc]

For the record, I've been trying to get this working in Sherlock, but can't really figure out how to pass username/password to yarn npm login. The shell trick (echo "$NEXUS_USER"; sleep 2; echo "$NEXUS_PASS") | yarn npm login which I use above doesn't seem to work when running in Sherlock.

[alamothe]

I found a solution in another issue. It is required to specify the token inside npmRegistries field. So your ~/.yarnrc.yml should look like:

npmRegistries:
  "https://my-registry-url":
    npmAuthToken: ...

npmScopes:
  myscope:
    npmAuthToken: ...

Not sure if both are required, but the first one certainly is.

[gustafc]

@alamothe this issue is called "yarn 3 ignores npmAuthTokens in ~/.yarnrc.yml" :) Running yarn npm login adds the token to ~/.yarnrc.yml, so that's not a problem; the problem is that yarn doesn't use the token from ~/.yarnrc.yml´ when I run yarn install`.

Don't know if adding npmScopeschanges anything, but we don't use scopes so that's an academic question. We use a Nexus server which both acts as proxy for the public NPM repo and serves our own private packages, no scopes involved, auth always required.

[alamothe]

@gustafc Interesting. It did ignore my token in ~/.yarnrc.yml if I only used npmScopes, but once I added npmRegistries too, it started to use it.

I definitely agree that there is a bug, I just wanted to help with a potential workaround until it is solved.

[arcanis]

It doesn't ignore the token, it just doesn't support configuration merging. In other words, the npmRegistries you declare in your project yarnrc overrides the one in the global one (same for npmScopes).

That's something we'll fix in the next major, since it's a breaking change, but it's quite tricky to implement in a satisfying way, so it hasn't been started yet.

[gustafc]

@arcanis I see. I eagerly await 4.0 then!

[alamothe]

Wow, so the reason it works for people who use scopes is that it can take npmScopes from project .yarnrc.yml and npmRegistries from home dir .yarnrc.yml. This way it is possible to separate scopes/registries configuration from tokens.

I don't see a workaround for people who don't use scopes.

@arcanis IMHO the only practical reason for separating the config is auth tokens, so perhaps that could simplify the solution i.e. it doesn't really need to be general.