feat: npm workspaces

[bajtos]

Rework the monorepo layout to use npm workspaces and per-component package.json files.

TODO:

• linters
• CI build
• Dockerfile(s)
• Dependabot

Documentation on different install strategies for npm workspaces:
https://docs.npmjs.com/cli/v10/commands/npm-ci?v=true#configuration

[bajtos]

I need to run standard for all files in the monorepo, including those that are not in any sub-package, e.g. migrations/index.js

[juliangruber]

Could we turn migrations into a sub-package too?

[bajtos]

Could we turn migrations into a sub-package too?

I would prefer indeed if both would work: linting from the parent and the subpackage. Could we create a standard config that re-exports the parent config?

I am not a fan of this. It adds a lot of boilerplate. We will need to add package.json to migrations, add scripts like lint, add standard as a dependency, and so on.

Having written that, I also want to get this monorepo setup finished ASAP and choose the setup that works for both of us, so I am going to take the direction you suggested.

[bajtos]

Could we turn migrations into a sub-package too?

Hmm, we would have to turn bin and scripts into sub-packages too. I definitely don't like that.

I am going to explore a different setup, where the monorepo root is a package on its own, and then some of the subdirectories are packages (workspaces) too.

[bajtos]

Done in daa2650

[bajtos]

I am keeping the lint-all step so that we can lint all workspaces in one go, and in parallel with running the tests.

The alternative is to lint spark-api and spark-publish as part of build-* workflows, and then add another workflow to run only files that are not part of any child workspace (bin, migrations, scripts). I am not sure how easy that would be to do, though, so I prefer the current proposal.

[bajtos]

In the main branch, one could run (cd spark-publish && npm test) to execute both the linter and the tests for a sub-package.

Is this something you want to preserve in the new monorepo setup?

In my proposal, you always have to run the linter from the monorepo root, because that's the place where package.json tells standard about Mocha.

[juliangruber]

I would prefer indeed if both would work: linting from the parent and the subpackage. Could we create a standard config that re-exports the parent config?

[bajtos]

Done in daa2650

[bajtos]

Could we create a standard config that re-exports the parent config?

I could not find any quick way how to do that, Standard seems to accept config only via CLI args or in the closest package.json file.

The alternative is to eject from standard to eslint, but that's much more work than I am willing to do now.

We could also remove the env config from package.json and modify every file to start with the following line:

/* eslint-env mocha */

It would be cleaner because non-test files won't have access to describe & it globals, but it also feels like too much boilerplate.

[bajtos]

I ended up duplicating the "standard":{"env":...}} section in each package.json.

[juliangruber]

This setup seems to work well for me - but I haven't stress tested it yet

[juliangruber]

Could we turn migrations into a sub-package too?

[juliangruber]

I would prefer indeed if both would work: linting from the parent and the subpackage. Could we create a standard config that re-exports the parent config?

[bajtos]

Dependabot's support for npm workspaces is a mixed bag. I guess we have to bite the bullet and hope all will work well.

• Dependabot not bumping dependencies in npm workspaces / monorepo structure dependabot/dependabot-core#5226
• Dependabot doesn't work properly with monorepos dependabot/dependabot-core#4993 (comment)
• Dependabot is not updating package-lock when bumping dependencies in a monorepo dependabot/dependabot-core#6346
• chore(deps): fix dependencies and dependabot config aws-powertools/powertools-lambda-typescript#1917
• Lockfile not updated for NPM workspaces using v3 lockfiles dependabot/dependabot-core#6270
• Issue with npm lock file v3 and sub modules: "npm ERR! npm ci can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with npm install before continuing." dependabot/dependabot-core#7441