[BUG] `npm install` creates unsync `package-lock.json`

[regseb]

Is there an existing issue for this?

• I have searched the existing issues

This issue exists in the latest npm version

• I am using the latest npm

Current Behavior

When I run npm install, the generated package-lock.json file isn't synchronized with the package.json file. The npm ci command fails. If I run npm install a second time: the package-lock.json file is modified (and synchronized).

Expected Behavior

npm install creates a package-lock.json file synchronized.

Steps To Reproduce

• Create package.json with this content:

  {
    "name": "testcase",
    "version": "1.0.0",
    "devDependencies": {
      "addons-linter": "6.13.0",
      "htmlhint": "1.1.4"
    }
  }

• npm install

  added 261 packages, and audited 262 packages in 27s
  
  62 packages are looking for funding
    run `npm fund` for details
  
  found 0 vulnerabilities
  

• npm ci

  npm ERR! code EUSAGE
  npm ERR! 
  npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
  npm ERR! 
  npm ERR! Invalid: lock file's node-fetch@2.7.0 does not satisfy node-fetch@2.6.11
  npm ERR! 
  npm ERR! Clean install a project
  npm ERR! 
  npm ERR! Usage:
  npm ERR! npm ci
  npm ERR! 
  npm ERR! Options:
  npm ERR! [--install-strategy <hoisted|nested|shallow|linked>] [--legacy-bundling]
  npm ERR! [--global-style] [--omit <dev|optional|peer> [--omit <dev|optional|peer> ...]]
  npm ERR! [--strict-peer-deps] [--foreground-scripts] [--ignore-scripts] [--no-audit]
  npm ERR! [--no-bin-links] [--no-fund] [--dry-run]
  npm ERR! [-w|--workspace <workspace-name> [-w|--workspace <workspace-name> ...]]
  npm ERR! [-ws|--workspaces] [--include-workspace-root] [--install-links]
  npm ERR! 
  npm ERR! aliases: clean-install, ic, install-clean, isntall-clean
  npm ERR! 
  npm ERR! Run "npm help ci" for more info
  
  npm ERR! A complete log of this run can be found in: /home/regseb/.npm/_logs/2023-09-10T14_22_27_045Z-debug-0.log
  

• cp package-lock.json package-save.json
• npm install

  changed 1 package, and audited 262 packages in 1s
  
  62 packages are looking for funding
    run `npm fund` for details
  
  found 0 vulnerabilities
  

• diff package-lock.json package-save.json

  1837,1839c1837,1839
  <       "version": "2.6.11",
  <       "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.6.11.tgz",
  <       "integrity": "sha512-4I6pdBY1EthSqDmJkiNk3JIT8cswwR9nfeW/cPdUagJYEQG7R95WRH74wpz7ma8Gh/9dI9FP+OU+0E4FvtA55w==",
  ---
  >       "version": "2.7.0",
  >       "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.7.0.tgz",
  >       "integrity": "sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A==",

The directory at the end with the files package.json, package-lock.json, package-save.json, and the directory node_modules/: testcase.zip

Environment

• npm: 10.0.0
• Node.js: v20.5.1
• OS Name: Ubuntu 22.04.3 LTS
• System Model Name: Dell Inspiron
• npm config:

; "user" config from /home/regseb/.npmrc

//registry.npmjs.org/:_authToken = (protected) 

; node bin location = /usr/bin/node
; node version = v20.5.1
; npm local prefix = /home/regseb/dev/testcase
; npm version = 10.0.0
; cwd = /home/regseb/dev/testcase
; HOME = /home/regseb
; Run `npm config ls -l` to show all defaults.

Related issues / pull request

• [BUG] npm can install an invalid tree with transitive optional peer deps #4859
• [BUG] npm install installs an invalid tree (as validated by npm ci) #5854
• [BUG] npm-ci failes if dependency with ^ has a new release that satifies #6378
• [BUG] Need to run npm install twice to fix npm ci command #7793
• [BUG] package-lock.json file inconsistent between different npm install runs #7841
• [BUG] npm dedupe step fails, a additional "npm i" run fixes the issue #7847
• [BC BREAK] version 11.6.2 breaks CI #8669
• [BUG] npm update generates package-lock.json that isn't considered in-sync by npm ci #8674
• feat(arborist)!: install optional peer dependencies #5301

[lukekarrys]

Triaged and this is happening on both npm 9 and npm 10.

[regseb]

I reproduce the bug with this package too:

{
  "name": "testcase",
  "version": "1.0.0",
  "dependencies": {
    "addons-scanner-utils": "9.3.0",
    "htmlhint": "1.1.4"
  }
}

• addons-scanner-utils has node-fetch 2.6.11 in peerDependencies (marked as optional);
• htmlhint has node-fetch ^2.6.2 in dependencies.

[siemhesda]

There is a weird thing that happens within the package.json that you may have missed out @regseb

When you do the first npm install, the package.json changes a bit from

{
  "name": "testcase",
  "version": "1.0.0",
  "devDependencies": {
    "addons-linter": "6.13.0",
    "htmlhint": "1.1.4"
  }
}

to

{
  "name": "testcase",
  "version": "1.0.0",
  "devDependencies": {
    "addons-linter": "6.13.0",
    "htmlhint": "1.1.4"
  },
  "dependencies": {
    "npm": "file:.."
  }
}

I am not sure if the second part is expected and should be added by npm. Otherwise, it still isn't the culprit on why the package-lock.json is not synced with package.json on the first npm install. Does this only happen with node-fetch @regseb?

@lukekarrys can you confirm that this issue happens on your end too? (Modification of the package.json)

[regseb]

There is a weird thing that happens within the package.json that you may have missed out @regseb

I tested with this two package.json and they aren't modified after npm install (with npm 10.1.0). And the package.json file in the testcase.zip (from my first message) hasn't been modified.

{
  "name": "testcase",
  "version": "1.0.0",
  "devDependencies": {
    "addons-linter": "6.13.0",
    "htmlhint": "1.1.4"
  }
}

{
  "name": "testcase",
  "version": "1.0.0",
  "dependencies": {
    "addons-scanner-utils": "9.3.0",
    "htmlhint": "1.1.4"
  }
}

---

Does this only happen with node-fetch @regseb?

This is the first time I've had this problem. I didn't check if I had other dependencies that were also loaded twice with different versions in dependencies and peerDependencies. But I don't think the problem is related to node-fetch.

[siemhesda]

There is a weird thing that happens within the package.json that you may have missed out @regseb

I tested with this two package.json and they aren't modified after npm install (with npm 10.1.0). And the package.json file in the testcase.zip (from my first message) hasn't been modified.

I noticed it is behaving so on my end since I was working on workspaces. In a normal set up it doesn't modify the package.json

Does this only happen with node-fetch @regseb?

This is the first time I've had this problem. I didn't check if I had other dependencies that were also loaded twice with different versions in dependencies and peerDependencies. But I don't think the problem is related to node-fetch.

I asked because all the examples you have given have node-fetch as dependencies or peerDependencies right?

[regseb]

I asked because all the examples you have given have node-fetch as dependencies or peerDependencies right?

I only have one case because addons-scanner-utils is a dependency of addons-linter:

testcase
+-- addons-linter 6.13.0
|   +-- addons-scanner-utils 9.3.0
|       +-- node-fetch 2.6.11 (peer)
+-- htmlhint
    +-- node-fetch ^2.6.2

My second example is just a version with one less intermediate step.

[allanpaiste]

TL;DR: this goes back to 7.0.9 which was the first release that caused the locked versions to be ignored on npm ci. The error Invalid: lock file's ... does not satisfy ... was introduced in 8.4.1 and is indeed correctly thrown as a guard against sneaky upgrades, but sneaky upgrades are actually caused by these two changes from the past (both still cause npm ci to misbehave):

• 7.0.9: 0e58e6f (npm ci installs newer version than what is in the lock).
• 8.6.0: bd96ae4 (npm ci thows the 8.4.1 assertion but would otherwise install newer version than what is in the lock).

Thus, the last glitch-free npm version is: 7.0.8

(As for the currently implemented installation error - it would serve much better as a test-case in integration test suite rather than user-facing functionality where it just causes confusion due to config-vs-lock being completely valid when it is thrown).

---

Our steps that get us to the error:

1. install package X with semver lock on a project Y: ^1.2.3 (new package-lock.json gets generated)
2. release new version of package X: 1.3.0
3. run npm ci for project Y.

EXPECTED: install still goes through and installs version 1.2.3 (as package-lock.json indicates)
ACTUAL: process errors out and informs that lock file's 1.2.3 does not satisfy 1.3.0

---

Some history:

This all goes way back to v7.X which got a bug introduced to it of which there have been attempts on fixing it ever since with varying level of success (the change in 8.4.1 being one of them).

• 8.4.0 - had an issue where when ^1.0.0 had produced locked version of 1.1.0 but latest available version in package index was 1.2.0 then 1.2.0 got installed (which violated the frozen version in lock).
• 8.4.1 - did not fix this but introduced an error that was thrown when this was attempted (in a sense it was not really a fix to the situation but rather a workaround that forced users to perform an upgrade to 1.2.0; problem: the same would repeat as soon as new version would be available).

The actual issue had been around longer and that specific release (and the error we have all been seeing) just represents an attempt to rein in the whole situation.

The hunt for the cause:

Though I initially focused on tracing down where the error originates from, it became since clear that instead of hunting for a release that does not fail the npm ci with the error - I should instead be searching for behaviour switch that the 8.4.1 code was catching. Little did I know that I'd be going back to the v7.X.

Below then are the findings by just going through the versions one-by-one to identify where the break happened (given in 3 steps where versions represent versions found in: package.json > package-lock.json > node_modules

• 7.0.8 - ^1.0.0 > 1.1.0 (lock) > npm ci > 1.1.0 (installed)
• 7.0.9 - ^1.0.0 > 1.1.0 (lock) > npm ci >1.2.0 (installed)

Thus the issue was somewhere in v7.0.8...v7.0.9.

From there, when doing another round of blind pin-pointing leads to this commit: 0e58e6f.

Starting from that specific commit, you'll start getting newer version installed than what is specified in the lock (which in turn in 8.4.1 was turned into a throw).

Most notably it's something about the change in build-ideal-tree.js

-      // didn't find a parent for it, but we're filling in external
-      // link targets, so go ahead and process it.
-      if (this[_follow] && !link.target.parent && !link.target.fsParent) {
+      // didn't find a parent for it or it has not been seen yet
+      // so go ahead and process it.
+      const unseenLink = (link.target.parent || link.target.fsParent)
+        && !this[_depsSeen].has(link.target)
+      if (this[_follow]
+        && !link.target.parent
+        && !link.target.fsParent
+        || unseenLink) {
         this.addTracker('idealTree', link.target.name, link.target.location)
         this[_depsQueue].push(link.target)
       }

Now when trying out newer version, I bumped into another "checkpoint" where same thing started to happen again (though this time around instead of getting a silent install, one would actually see the infamous error that is stated at the original report of the issue).

The second change was also done to build-ideal-tree.js, this time within the 8.6.0 release.

       .then(tree => {
+        // search the virtual tree for invalid edges, if any are found add their source to
+        // the depsQueue so that we'll fix it later
+        depth({
+          tree,
+          getChildren: (node) => [...node.edgesOut.values()].map(edge => edge.to),
+          filter: node => node,
+          visit: node => {
+            for (const edge of node.edgesOut.values()) {
+              if (!edge.valid) {
+                this[_depsQueue].push(node)
+                break // no need to continue the loop after the first hit
+              }
+            }
+          },
+        })
         // null the virtual tree, because we're about to hack away at it
         // if you want another one, load another copy.

Sadly enough, neither of the changes have proper test coverage which does not really inject confidence to the reader in terms of knowing that the base functionality of the installer remains the same (second change just modifies existing tests which just shadows potential new bug).

As such, would appreciate if someone else would take it from here.

---

Conclusion:

1. Reverting the change 0e58e6f will make npm ci behave expectedly up to version 8.5.5.
2. Reverting the change bd96ae4 (together with the one above) will make npm ci behave expectedly up to version 10.2.0 (latest).

Whatever these two changes claimed to fix - they introduced new issues. So those fixes should be re-introduced with proper test-coverage or re-reported as issues after reverting the broken fixes (I'd rather have those bugs back than the main intent of lock file being broken).

[allanpaiste]

Perhaps @isaacs @nlf @lukekarrys can comment on the changes mentioned above and what they're actually addressing. Could these pieces be re-imagined with provided unit/integration tests that would both guard the pre-7.0.9 functionality and also prove that the changes fix whatever they were supposed to fix.