Make bsb able to compile all third-party artifacts into the top-level lib/

[chenglou]

Folks might accidentally use npm link or npm install local/path and have npm symlink the dependency into their own project, either for testing or for whatever other purposes. This is a common scenario and because bsb places artifacts inside these symlinked dirs, some weird behaviors happen.

I think it's reasonable to provide an option to ask bsb to output artifacts outside of the third-party dirs. Maybe in the top-level lib/js folder. Maybe we can pass it as a CLI flag while we're at it so that we don't need to care where to place the artifacts. This allows us to interop with other package managers and workflows better without too much cost.

It'd be a config from the top-level that overrides whatever the third-party bsconfig says.

cc @bsansouci who needed this

Edit: the flag approach might be better, e.g. bsb -artifacts-location foo/bar (is ninja ok with this?). We might need flags soon anyway if we want to have a dev vs prod mode (for e.g. printing debugging?)

[jordwalke]

@bsansouci

In addition to making bs-platform(JS) work with npm/yarn link, this is the blocker that will allow bs-platform(JS) to work with esy. esy requires that all package builds support symlink workflows, and so requires that either the top level app maintain all its build artifacts in its own tree (like this issue proposes), or that each individual package place its artifacts into the carefully managed install destination provided by esy. Either way will work. bsb-native will already place artifacts into the esy install destination by creating a .install file, but bs-platform(JS) does not, and it would be hard to make bs-platform install into the correct location for other reasons. So while bsb-native might use .install, for bs-platform(JS) the easiest solution is to have the top level app place artifacts into lib for all dependencies.

Longer term, it would be beneficial to have bs-platform(JS) follow the style of bsb-native and place artifacts into a carefully managed esy cache that is compatible with symlink workflows, but until it does so, having bs-platform(JS) place all artifacts for dependencies into the top level app's ./lib directory not only fixes npm link workflows today, but also makes bs-platform compatible with esy today.

[bsansouci]

I'd love to see this happen! I'll start experimenting with this in bsb-native this week. I'll tag you on the changes :)
If @bobzhang is ok with it, I can make a PR to bsb.

[mlms13]

I'd love to see this happen!

My use case: I'm building a node app (so no bundler like webpack). I have a Reason library as a dependency. When the library is compiled, its artifacts end up in node_modules/ (which I haven't checked in) instead of lib/ (which I have checked in). When the deployment to my server does a fresh npm install, it can't resolve the compiled js modules.

I worked around this by making the deploy script responsible for installing bsb and compiling on npm's postinstall, but this slows down my deployment (because bs-platform used to be a devDependency, but now it needs to be a real dependency).

[jordwalke]

Yarn is moving to an esy style cache of dependencies not located in node_modules and it appears that npm is also experimenting with this mechanism as well. In Yarn's model writing into node_modules is not an option either, so it looks like bsb should think about solving this soon.

[chenglou]

Yeah agreed

[TheSpyder]

Possibly relevant to this, if dependencies bs-platform compiles are symlinked there are issues - see #1691

This might be a nice workaround for the stdlib requests in #2127, although for my use case the separate stdlib is still essential.

[bobzhang]

it is okay to move .cm[ij] files into lib, but it is complex for js files