Reliably determine @rescript/runtime path #7637

cknitt · 2025-07-12T06:18:11Z

(updated description after moving the runtime to @rescript/runtime and deciding to use a command line arg instead of an env var)

To determine the path of the runtime module @rescript/runtime in a reliable way that works with all package managers / package directory layouts, we need to use the JS module resolution algorithm.

In rewatch, we can use https://github.com/oxc-project/oxc-resolver.
For the legacy build system, we can just use require.resolve.

This should fix all issues with pnpm not finding the runtime module.

pkg-pr-new · 2025-07-12T06:26:30Z

Open in StackBlitz

rescript

npm i https://pkg.pr.new/rescript-lang/rescript@7637

@rescript/darwin-arm64

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/darwin-arm64@7637

@rescript/darwin-x64

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/darwin-x64@7637

@rescript/linux-arm64

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/linux-arm64@7637

@rescript/linux-x64

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/linux-x64@7637

@rescript/runtime

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/runtime@7637

@rescript/win32-x64

npm i https://pkg.pr.new/rescript-lang/rescript/@rescript/win32-x64@7637

commit: 3683436

cknitt · 2025-07-12T07:15:35Z

BTW this approach should also make it easier to move the runtime to @rescript/runtime.

cknitt · 2025-07-12T10:50:55Z

@zth This probably needs to be taken into account in the editor extension, too?

BTW this approach should also make it easier to move the runtime to @rescript/runtime.

So maybe the variable should actually be RESCRIPT_RUNTIME or RESCRIPT_RUNTIME_PATH or something like that.

zth · 2025-07-12T18:17:20Z

@cknitt this means that you have to supply that env variable always to be able to run bsc standalone...?

cknitt · 2025-07-12T18:55:51Z

Yes. I mean we could fall back to the previous algorithm if the variable is not set, but we need to be aware that there are cases where this algorithm will not work.

We could also use a command line argument instead of an env var if that's your concern - just more work to implement to get it passed through for both rewatch and bsb.

nojaf · 2025-07-12T19:13:00Z

Would like to have a fallback for this. I frequently run bsc.

cknitt · 2025-07-13T05:08:35Z

Note that that would only be an issue when directly running the platform-specific bsc.exe though.

When running bsc via yarn bsc or ./node_modules/bin/bsc, you would be running the node wrapper script that correctly supplies the env var.

nojaf · 2025-07-13T07:36:16Z

Note that that would only be an issue when directly running the platform-specific bsc.exe though.

That includes running dune exec bsc right? If so, could you please add a note which relative path to set from this repository.

zth · 2025-07-13T09:25:46Z

Would it be possible to make it an optional bsc arg (that uses the old algorithm if not supplied) and bake resolving that into the "compiler args" command? Rewatch could call a node script to get the path. Maybe in the future we could check if this resolution is needed at all by checking what package manager is used.

Bsc is already used standalone in a few places, and I envision it being used even more standalone in the future (Svelte integration, one off scripts, code blocks in docs etc). The overhead of calling a node process for each invocation is non-negligible, and the orchestration for figuring out how to call it shouldn't be too difficult (that's why compiler args is so nice).

cknitt · 2025-07-14T06:09:59Z

Yes, we could make it an optional bsc arg. There actually already exists an optional arg -runtime for the current "external stdlib" feature. To reduce confusion, maybe we should complete the work started in #7483 first. So that we get rid of @rescript/std, and the stdlib always comes from @rescript/runtime. The -runtime flag would then be (optionally) used to pass the location of that package to handle the various non-standard node_modules layouts.

If the flag is not set, the default could be the simple implementation we had earlier that just assumed npm's node_modules layout.

I would avoid rewatch calling node though, I'd prefer node/bun/whatever calling rewatch via the wrapper script, with the wrapper script passing the correct location of the runtime as implemented in this PR. rewatch can then directly call bsc.exe with the -runtime flag, or include the -runtime flag in the compiler-args output.

cknitt · 2025-09-01T07:51:28Z

Now using a -runtime-path arg in bsc, and oxc_resolver in rewatch to determine the path of @rescript/runtime.

zth · 2025-09-01T07:56:22Z

@cknitt don't know the details here, but a thought - would "compiler args" contain everything needed for the runtime path? So that one can always call that and get everything needed to call bsc.

Maybe something obvious, but thought I'd ask.

cknitt · 2025-09-01T08:11:40Z

Yes, compiler args will also contain the -runtime-path! 🙂

nojaf

Still a little on the fence.

nojaf · 2025-09-01T08:03:00Z

cli/bsc.js


 const delegate_args = process.argv.slice(2);
+if (!delegate_args.includes("-runtime-path")) {


So this happens when we run bunx bsc directly. It makes me wonder if we could just reuse this and pass it to Rewatch via an environment variable.

I'm not against oxc_resolver per se, but doesn't this create two different mechanisms for resolving the -runtime-path?

Passing the runtime path to rewatch via an env var or command line flag was my original intention, but then I thought it was better if rewatch can remain self-contained here so that you can just invoke the binary directly and it does the resolution itself.

Well, my experience with calling rewatch directly is that I need to set RESCRIPT_BSC_EXE, so it seems a bit weird that we do this for the runtime but not the compiler.

cli/common/runtime.js

nojaf · 2025-09-01T08:10:44Z

rewatch/src/helpers.rs

@@ -249,26 +250,41 @@ pub fn get_bsc() -> PathBuf {
        .to_stripped_verbatim_path()
 }

-pub fn get_rescript_legacy(project_context: &ProjectContext) -> PathBuf {
+pub fn get_runtime_path(project_context: &ProjectContext) -> anyhow::Result<PathBuf> {
+    let resolver = Resolver::new(ResolveOptions::default());


Is the resolver expensive to make?
If this helper called multiple times, it might make sense to have this outside the function and scoped to the lifetime of the process.

I don't know, but it is only called once in initialize_build if I am not mistaken.

Okay, we should be mindful of it if we end up using it again for other parts.

There is a part of hoping it can be used in helpers::try_package_path as well (used by packages::read_dependency and config::flatten_ppx_flags).
In there, we are trying some logical things, yet perhaps we can offload it as well to oxc.

nojaf · 2025-09-01T08:26:52Z

rewatch/src/build/build_types.rs

@@ -97,6 +97,7 @@ pub struct BuildState {
    pub module_names: AHashSet<String>,
    pub deleted_modules: AHashSet<String>,
    pub bsc_path: PathBuf,
+    pub runtime_path: Option<PathBuf>,


Not 100% sold that this should be part of the BuildState.
Computing it in compile::compile is probably fine.
The fact that is is optional is a bit of a smell to make.
Clean doesn't use this.

nojaf · 2025-09-01T08:28:29Z

rewatch/src/build.rs

@@ -89,6 +89,9 @@ pub fn get_compiler_args(rescript_file_path: &Path) -> Result<String> {
        interface_filename.push('i');
        PathBuf::from(&interface_filename).exists()
    };
+
+    let runtime_path = helpers::get_runtime_path(&project_context).ok();


Should this not fail? Isn't this destined to throw a more cryptic error in bsc if it wasn't found?

Well, it falls back to the default (simple) implementation in bsc then, but it can still continue.

When building @rescript/runtime itself, the resolution will fail for example.

(.ok means "toOption", weird naming...)

Might be good to add that somewhere in the code as well, when it is fine to be None.

nojaf · 2025-09-01T08:31:29Z

rewatch/src/helpers.rs


-    rescript_legacy_path.unwrap_or_else(|_| panic!("Could not find rescript-legacy.exe"))
+    // Resolve <project> -> rescript


Could you elaborate in a code comment why you are resolving rescript package first? This seems like an unnecessary extra step. Why not search for @rescript/runtime from the get go?

Tried that, but didn't work, probably because there is no direct dependency on the runtime package. Or maybe I was doing something wrong. I can have another look.

No — that may just be how resolve_oxc and https://nodejs.org/api/esm.html#resolution-algorithm work. That’s fine, but we should document it.

There’s a tendency to assume oxc_resolver can solve everything; I’m all for leaning on it when it helps, but we need to understand and document what it can and cannot do.

Let’s leave clear "whys" for future contributors so they understand our decisions and don’t keep getting challenged.

rewatch/src/helpers.rs

cknitt changed the title ~~Stdlib dir~~ Reliably determine standard lib dir Jul 12, 2025

cknitt force-pushed the stdlib-dir branch from b864763 to 373c19a Compare July 12, 2025 06:20

cknitt requested a review from zth July 12, 2025 06:32

cknitt mentioned this pull request Jul 12, 2025

Fix finding the standard library for pnpm some more #7624

Closed

cknitt requested a review from cristianoc July 12, 2025 06:36

cknitt force-pushed the stdlib-dir branch 2 times, most recently from a18e180 to 2daa984 Compare July 12, 2025 13:27

cknitt force-pushed the stdlib-dir branch 2 times, most recently from b7afc8e to 56bb638 Compare July 13, 2025 05:36

cknitt marked this pull request as draft July 14, 2025 05:54

This was referenced Aug 26, 2025

Remove @rescript/std (in favor of @rescript/runtime) #7811

Merged

12.0.0-alpha.13: Pervasives not found when using pnpm #7526

Open

cknitt force-pushed the stdlib-dir branch 2 times, most recently from d64d7ed to 0ccec74 Compare August 31, 2025 18:10

cknitt changed the title ~~Reliably determine standard lib dir~~ Reliably determine @rescript/runtime dir Aug 31, 2025

cknitt changed the title ~~Reliably determine @rescript/runtime dir~~ Reliably determine @rescript/runtime path Aug 31, 2025

cknitt added 3 commits September 1, 2025 08:31

Move all rescript legacy js sources to rescript-legacy folder

9d30978

Remove unused rewatch.cmd

a736b6d

New module Runtime_package

bad2fa9

cknitt force-pushed the stdlib-dir branch from 0ccec74 to 709b06a Compare September 1, 2025 06:54

cknitt force-pushed the stdlib-dir branch from 709b06a to 1817c77 Compare September 1, 2025 07:14

cknitt added 5 commits September 1, 2025 09:18

bsc: add -runtime-path option

fc8dead

Rewatch determine runtime path via oxc_resolver and pass to bsc

27317d9

Add -runtime-path handling to bsc.js

fda75a6

Perform installation test with new build system

d409a79

Also resolve rescript legacy via oxc_resolver

f6d8303

cknitt force-pushed the stdlib-dir branch from 1817c77 to f6d8303 Compare September 1, 2025 07:19

nojaf reviewed Sep 1, 2025

View reviewed changes

rewatch/src/helpers.rs Show resolved Hide resolved

Use ESM way to determine runtime path

8f9dfec

cknitt force-pushed the stdlib-dir branch 3 times, most recently from 1b65720 to 944d91b Compare September 1, 2025 10:35

cknitt added 3 commits September 1, 2025 12:50

Set -runtime-path in legacy build (bsb), too

1bacfda

Also test legacy build in installation test

a726fa0

Log warning if runtime path couldn't be determined

3683436

cknitt force-pushed the stdlib-dir branch from 944d91b to 3683436 Compare September 1, 2025 10:50

cknitt mentioned this pull request Sep 1, 2025

V12 beta doesn't work with Deno #7828

Open

5 tasks


		const delegate_args = process.argv.slice(2);
		if (!delegate_args.includes("-runtime-path")) {


		rescript_legacy_path.unwrap_or_else(\|_\| panic!("Could not find rescript-legacy.exe"))
		// Resolve <project> -> rescript

Reliably determine @rescript/runtime path #7637

Are you sure you want to change the base?

Reliably determine @rescript/runtime path #7637

Uh oh!

Conversation

cknitt commented Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new bot commented Jul 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cknitt commented Jul 12, 2025

Uh oh!

cknitt commented Jul 12, 2025

Uh oh!

zth commented Jul 12, 2025

Uh oh!

cknitt commented Jul 12, 2025

Uh oh!

nojaf commented Jul 12, 2025

Uh oh!

cknitt commented Jul 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nojaf commented Jul 13, 2025

Uh oh!

zth commented Jul 13, 2025

Uh oh!

cknitt commented Jul 14, 2025

Uh oh!

cknitt commented Sep 1, 2025

Uh oh!

zth commented Sep 1, 2025

Uh oh!

cknitt commented Sep 1, 2025

Uh oh!

nojaf left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cknitt Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cknitt commented Jul 12, 2025 •

edited

Loading

pkg-pr-new bot commented Jul 12, 2025 •

edited

Loading

cknitt commented Jul 13, 2025 •

edited

Loading

cknitt Sep 1, 2025 •

edited

Loading