Add an option to warn when importing duplicate versions of the same module

[philipwalton]

There are several webpack plugins that offer similar functionality (e.g. this one), but IMO warning about duplicates should be a built-in feature of any bundling tool.

In a rollup build today I noticed I was seeing multiple instances of the same classes in my bundle (e.g. WorkboxError, WorkboxError$1, WorkboxError$2, etc.), so I added the following code to my build script to detect duplicates:

const moduleIds = {}
for (const module of bundle.modules) {
  const NODE_MODULES = 'node_modules';

  if (module.id.includes(NODE_MODULES)) {
    const id = module.id.slice(
        module.id.lastIndexOf(NODE_MODULES) + NODE_MODULES.length + 1);

    if (!moduleIds[id]) moduleIds[id] = [];

    moduleIds[id].push(module.id);
  }
}

for (const [id, instances] of Object.entries(moduleIds)) {
  if (instances.length > 1) {
    console.log(`Duplicate module detected for: ${id}`)
    console.log(instances);
  }
}

After that, I get lots of entries like the following one reporting duplicate modules:

Duplicate module detected for: workbox-core/_private/WorkboxError.mjs
[ '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-background-sync/node_modules/workbox-core/_private/WorkboxError.mjs',
  '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-routing/node_modules/workbox-core/_private/WorkboxError.mjs',
  '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-strategies/node_modules/workbox-core/_private/WorkboxError.mjs',
  '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-broadcast-cache-update/node_modules/workbox-core/_private/WorkboxError.mjs',
  '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-precaching/node_modules/workbox-core/_private/WorkboxError.mjs',
  '/Users/philipwalton/Projects/philipwalton/blog/node_modules/workbox-cache-expiration/node_modules/workbox-core/_private/WorkboxError.mjs' ]

Note that these aren't different versions of the same module, they're actually the same version, they're just imported multiple times because they have different paths (i.e. they're in nested node_module directories rather than a common top-level directory shared by all consuming modules).

[tivac]

Does a npm dedupe solve it?

Not that a warning wouldn't be nice, just curious if this could be solved in the near-term by cleaning up your node_modules folder a bit.

[philipwalton]

npm dedupe can solve it in many cases, but with a bit of trying I was able to put together a case that npm dedupe didn't solve.

My main concern is not my specific case (I've already temporarily fixed that), but more for people who don't take the time to look at their bundles or don't understand how/why duplication can occur.

[enure]

I have seen this happen when you are either using yarn link or file: to link to local dependencies where these local dependencies also have a node_modules dir.

rollup-plugin-node-resolve will then cause module duplication because it finds the same module in the main node_modules dir and as well in dependent node_modules dirs. In this case the paths are different so it thinks they're unique modules.

I initially filed an issue against rollup/rollup, rollup/rollup#2425.

The fix that's working for us is:

resolve({
  customResolveOptions: {
    moduleDirectory: path.join(process.cwd(), 'node_modules')
  }
})

I have a project which repros the issue I'm describing above here:
https://github.com/enure/rollup-codesplit-preact-duplication

[sormy]

This issue is duplicate of #80

[sormy]

I added submitted dedupe PR to address this issue: #201

[dima-takoy-zz]

PR was merged. Close it too ;-)

[philipwalton]

Hmmm, that PR doesn't actually solve this issue. As far as I can tell, it won't discover duplication and warn the developer. See my comment here: #201

[sormy]

My 5 cents.

I think warning about module duplicates should not be implemented on rollup-plugin-node-resolve level since different plugins could provide resolveId hook and implementing feature on the level of this plugin won't help to identify duplication caused by resolveId hook from any other plugin.

[philipwalton]

@sormy yes, I agree with this.

Regardless of where the check lives, my opinion is the default behavior should be to error when duplication is detected and developers who explicitly need the behavior can disable the error.

[bterlson]

Closing this as I think there's consensus this plugin isn't the right place for a duplicate modules check.

I'll also note that duplicate modules due to symlinking should be drastically reduced with some recent improvements.