Infer project references from common monorepo patterns / tools

[RyanCavanaugh]

Genesis: see section "Repetitive configuration" in #3469 (comment)

Search Terms

monorepo infer project references automatically yarn lerna workspace package.json

Suggestion

For common monorepo managers, we should natively understand cross-project references declared in package.json as if they were declared in tsconfig.json

Open questions:

• Which formats (lerna, yarn, pnpm, etc) would be supported? Can all of them be consistently detected, or would you need to opt in to a specific "monorepo format" to enable a specific resolution algorithm?
• How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?
• Would you need to opt in? Would there be a way to opt out? What should that look like?

Use Cases

• Monorepos of all (supportable) flavors

Examples

https://github.com/RyanCavanaugh/learn-a

This repo has a fair bit of duplication where projects need to write down their dependencies in package.json and as references (with different syntax) in tsconfig.json.

Checklist

My suggestion meets these guidelines:

• This wouldn't be a breaking change in existing TypeScript / JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. new expression-level syntax)

[weswigham]

Which formats (lerna, yarn, pnpm, etc) would be supported? Can all of them be consistently detected, or would you need to opt in to a specific "monorepo format" to enable a specific resolution algorithm?

All the tools you've listed use normal commonjs module resolution (and consequently use the normal package.json dependencies, optionalDependencies, and to a lesser degree devDependencies fields) - none of them change runtime behavior at at all in that regard. Where they differ is where they symlink things into/from (some install all packages in one dir, then link them into each other; some install everything in a top-level node_modules, since that makes everything available), which is what causes issues for us, usually - our symlink handling needs to be essentially perfect to handle all of these configurations correctly. All that'll solve that is hammering away at it with all the configurations we can find and fixing the bugs we find, IMO.

One of the things I was running into in trying to setup an ui-fabric user suite test (where I symlinked things together in our harness rather than use rush) was that depending on where I symlinked stuff, TS would locate modules in ways I wasn't always prepared for. For example: If I symlink from packages/foo into packages/bar/node_modules, imports from packages/foo/index.d.ts is still going to resolve up from packages/foo, not packages/bar/node_modules/foo (which complicates where you need to place things a bit). These tools all usually handle all these cases right (they have to or the runtime wouldn't work); we just need to continue to be faithful to the commonjs resolver's behavior.

How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?

I would assume package root, and if we don't find one there, we could read a tsconfig field in the package.json that points at it (cue people asking us to read the configuration straight from the package.json).

Would you need to opt in? Would there be a way to opt out? What should that look like?

The usecase is a (near) zero-config start for common monorepo setups, so opt-out, IMO. Could add something like a --no-pkg flag to tsc -b that stops it from walking package.json files.

[bajtos]

How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?

In https://github.com/strongloop/loopback-next, we are using monorepo to develop a bunch of modules. Right now, we have one tsconfig.json file per each package. (Because of the way how we are working around missing support for project references, we call this file tsconfig.build.json.)

IMO, it's important to allow each package to have its own tsconfig configuration, because this configuration often involves a list of files to include/exclude from compilation.

Assuming the tsconfig.json file is located in the package root makes perfect sense to me 👍

Initially, I was reading the proposal as to assume a single tsconfig.json file located in monorepo root, that would not work (at least for us).

Would you need to opt in? Would there be a way to opt out? What should that look like?

If we can find an elegant way how to support most of commonly-used monorepo layout (tools) that are setting up cross-package dependency tree using the information from standard npm/package.jsondependencies, optionalDependencies and devDependencies fields only, then I think TypeScript's build mode should automatically infer project references from package.json too.

I think the tricky question we need to answer first: how to distinguish between dependencies that are considered as monorepo-local and should be configured as TypeScript references; and external dependencies that should be consumed as read-only? The package.json file does not provide any hints on that.

The first solution that comes to my mind:

• Find out where is the monorepo rooted, either by looking for lerna.json, yarn config file, etc. or by asking the user to explicitly provide that directory via configuration. I think the explicit config option is a better solution as it does not couple TypeScript with different monorepo solutions and provides a natural place for opting into auto-discovery of project references.
• When parsing package.json dependencies, check out which dependencies are resolved as a symlink pointing to a different place inside monorepo - these dependencies should be added as project references. Dependencies symlinked to a different place (outside of monorepo, typically /user/local/lib/node_modules when using npm link manually) (*) or installed as a copy in node_modules should be consumed as read-only.

I find this solution a bit too complex and involved, but don't have any better alternative right now :(

(*) Handling of symlinks outside of monorepo is actually another interesting case to discuss. If we treat all symlinked dependencies as project references, then people using multiple single-repo projects with manual npm link could get benefits from the new build mode too.

For example, loopback depends on strong-remoting, where both modules are maintained by the same team. Sometimes I need to make changes both in loopback and strong-remoting, where the change in loopback depends on the changes made in strong-remoting. To do that, I run npm link path-to-my-strong-remoting-clone in my loopback directory. If we were using TypeScript and the build mode was treating all symlinks as project references, then I could make cross-project renames and rebuild both loopback & strong-remoting in a single build step.

[weswigham]

I think the tricky question we need to answer first: how to distinguish between dependencies that are considered as monorepo-local and should be configured as TypeScript references; and external dependencies that should be consumed as read-only?

I was thinking we could update the -b flag to accept a list of globs (whereas today it takes a single folder or list thereof), so it works like the packages field in lerna or workspace fields in npm/yarn, so you can easily pass tsc -b packages/* apps/* to say "all the packages in these two folders are part of my build". I've found that to be the concise way to express how I've seen these repos laid out when I've been comparing rush, learna, and workspaces - of those, rush is the only one that I've seen be regularly much more explicit than that in its config. IMO, glob support in the input paths are probably worthwhile even if you'd not use package.json reading.

[mhegazy]

how would that work for tsserver?

[weswigham]

@mhegazy AFAIK Nothing in tsserver yet cares about project references (or the dependency graph thereof), just the presence of declaration maps. 😉 The only one that requires it is probably a cross-project compile on save - which is a matter of integrating the entire -b flag into the server - arguments included in some way. Which, for that, we could choose to interpret a top-level tsconfig similar to

{
  "references": [{ path: "packages/*" }]
}

to setup the context. (Which, hopefully, would also allow tsc -b in that directory to work without any arguments). This mimics a lerna.json, a rush,json, or the top-level package.json used for npm/yarn workspaces.

[weswigham]

I mean, jamming it into a tsconfig is ultimately wrong (a .tsbuildrc.json with dedicated build-context wide settings would be more appropriate), since tsc -b's arguments don't correspond to tsc's normal arguments at all, so re-purposing tsc's config file is also wrong (since 99% of it would be meaningless and the remaining 1% would be repurposed overlap), but I've lost that fight already.

[weswigham]

Or alternatively, we can just actually read a lerna.json, a rush.json (maybe), or a top-level package.json with a workspaces field; because why duplicate even that configuration when you don't need to.

[Cryrivers]

I think cross repo reference + yarn workspace worked before TypeScript 2.9

Say you want to import something from your monorepo folder packages/@common/library，the code fix feature in VSCode could correctly suggest the path @common/library in TypeScript 2.8.

However, since 2.9, I believe it goes through symlinks or something, VSCode now suggests something like ../../../../../../node_modules/@common/library, resolving all the way back to the node_modules folder in monorepo root (with yarn workspace enabled).

[timfish]

Also bear in mind that the various tools can treat the workspace globs differently.

I tried pnpm recursive install which goes looking through every subdirectory whereas other monorepo tools I've tried only look at the first level of directories. That meant it also went through a jspm_packages directory we have and tried to install and symlink all those too!

The downside to trying to support specific monorepo types is that these are just the ones we're using this week. Next week we'll probably be using something completely different 😆

[friflo]

I am using a monorepo setup based on lerna. The problem I have in this scenario is, that the root folder contain all dependencies of all packages.
This means that every *.ts file is able to import any module, which is in root/node_modules.
Before I made this experience I expected that only the package.json dependencies are considered to be resolved. This is what a developer would expect.
I think it is sufficient to apply this type of module resolution at compile time via a config setting in tsconfig.json (e.g. "dependencies": "./).
The module resolution at runtime (via node) should be unchanged.