[BUG] npm can install an invalid tree with transitive optional peer deps

[billyjanitsch]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

Given the following packages:

{
  "name": "app",
  "version": "1.0.0",
  "dependencies": {
    "a": "^1.0.0",
    "b": "^1.0.0"
  }
}

{
  "name": "a",
  "version": "1.0.0",
  "peerDependencies": {
    "c": "^1.0.0"
  },
  "peerDependenciesMeta": {
    "c": {
      "optional": true
    }
  }
}

{
  "name": "b",
  "version": "1.0.0",
  "dependencies": {
    "c": "^2.0.0"
  }
}

{
  "name": "c"
  "version": "1.0.0"
}

{
  "name": "c"
  "version": "2.0.0"
}

Running npm install in app completes successfully, but results in the following invalid tree (npm ls):

app@1.0.0
├─┬ a@1.0.0
│ └── c@2.0.0 invalid: "^1.0.0" from node_modules/a
└─┬ b@1.0.0
  └── c@2.0.0 deduped invalid: "^1.0.0" from node_modules/a

Basically, b's version of c gets incorrectly hoisted, resulting in an incompatible peer dep for a.

npm ci also refuses to install the tree that npm install just generated:

npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.

Related to #4664. That issue was closed but there does seem to be an npm bug here.

Expected Behavior

I haven't thought through the details carefully, but it seems that b's dependency on c shouldn't be hoisted if it would result in an invalid optional peer dep.

Steps To Reproduce

This reproduces in npm 8.6.0 through 8.9.0.

1. Clone https://github.com/billyjanitsch/npm-optional-peer-deps-hoist
2. Note the two separate commits. The first commits the package.json, and the second commits the package-lock.json generated by npm install.
3. Run npm ci or npm ls; both exit with an error.

@pmmmwh/react-refresh-webpack-plugin@0.5.5 optionally peer-depends on type-fest@>=0.17.0 <3.0.0. ow directly depends on type-fest@^0.11.0. The latter gets incorrectly hoisted.

Environment

• npm: 8.9.0
• Node.js: 16.15.0
• OS Name: macOS
• System Model Name: MacBook Pro
• npm config:

; n/a

[billyjanitsch]

(cc @lukekarrys, per your comment in #4664 (comment).)

[bnb]

I think I'm also hitting this on 8.9.0 and it's causing a Dockerfile build to fail, though it's not restricted to peerDeps - all my deps are failing to install with npm ci.

[bnb]

created a Gist of my package.json, package-lock.json, and log file: https://gist.github.com/bnb/f05ed6601c7997c0453d062ac7c23a64

I've nuked node_modules and run npm install and then npm ci --only=production again, get the same error output. I've also just run npm install without nuking node_modules and run npm ci --only=production and also get the same error output.

[billyjanitsch]

@bnb I don't think yours is the same issue. They share the same error message ("npm ci can only install packages...") but the failure state is different. In my case, npm install has produced an invalid tree/lockfile due to an interaction between peer deps and hoisting, and npm ci is (correctly) realizing that the tree is invalid and refuses to install.

I don't believe that your case is a bug, though the error message could be improved. You generally need to run npm install with the same args/config as npm ci in order for the latter's lockfile validity check to work correctly. On 8.9.0, starting from your package.json and package-lock.json, if I run npm install --only=production && npm ci --only=production, both installs complete without error. (If you can't reproduce this, I'm curious if there's anything in your config (npm get) that might be altering the lockfile generation.)

The error message should probably make clear that npm install needs to be run with the same config as npm ci, but it might be best to note that in a new issue since that's not a factor in this one.

[darcyclarke]

@billyjanitsch Thanks for the thorough report & repro! I was able to reproduce (on v8.5.5 & v8.9.0 -- so this is notably not related to v8.6.0 specifically).

Funny enough, I was able to run npm install a second time & that resolved the problem for me (so likely a race condition &/or missing condition in canPlaceDep when we're building the ideal tree from scratch -- ie. without a lockfile or node_modules).

How to unblock yourself right now:

• remove node_modules/package-lock.json
• run npm install
• run npm install (again)
• run npm ci or npm ls --all should now succeed (& the lockfile should be accurate)

The issue is as you explained it; the direct dep gets hoisted the first time we reify but it conflicts with the optional peer. Running install a second time (w/ a lockfile) will actually correctly nest the direct dep & hoist the peer (updating the lockfile in-turn & hopefully avoiding future problems).

We'll definitely look into where to solve this early next week.

[billyjanitsch]

Thanks @darcyclarke. I mistakenly assumed that this didn't occur on <8.6.0 because I was using the npm ci failure as a test, but ofc it won't fail prior to 8.6.0 because that's when the check was added, even if the tree is invalid in this way. Thanks for looking into it and suggesting a stopgap workaround. 🙂

[darcyclarke]

All good, we'll get a fix in soon.