Re-enable Scala Native (0.5.8); keep ZipOps JVM-only; split zip tests to JVM

[rishi-jat]

• Enable Native tests with nativeLinkStubs=true; update build.mill
• Move ZipOps to os/src-jvm; add shared placeholder for Native
• Move zip/unzip tests to os/test/src-jvm; add CheckerZipTests and Native placeholder suite
• Stabilize FilesystemMetadataTests.isExecutable
• Docs: README note on JVM-only Zip APIs and changelog

Fixes #395

[rishi-jat]

/cc @lihaoyi

[rishi-jat]

@lihaoyi can you please review this PR.Thanks!

[LeeTibbert]

Life is strange! If I read the failing log files correctly, the failures appear to be Process or SubProcess related.
The particular failing tests seem to vary.

The first os-lib CI run failed only on one Ubuntu version, the latest fails on both Ubuntu versions and
both macOS.

Windows succeeds which is a pleasant surprise due to the fact that SN PR #4367 Windows only: sys.process.Process hangs since v0.5.7 is unresolved in SN 0.5.8 (and in current 0.5.9-SNAPSHOT). I am aware that this more probably means
that no test was run which would have triggered the bug rather than that the bug is fixed.

To help me figure out what might be failing:

• Do a different set of failures occur if os-lib CI is run again?

• Do the Process and Subprocess tests succeed when run locally on your system?
  Does the whole ensemble need to be run to provoke the failure or can running just,
  say SubProcess tests, show the failure?

I may have to copy os-lib with this PR down and build locally to me, probably with
SN 0.5.9-SNAPSHOT.

[LeeTibbert]

The current ubuntu-latest, 11 failure is in SubprocessTests.envArgs.
That has a number of "locally" sub-tests. In my sandbox environment, is
there any way, short of hacking, to tell which of the sub-tests failed?

It is very early days, nay hours, but from the log files, the unifying theme behind
the possible SN errors is that a comparison between the character stream returned
by a SubProcess does not match a fixed character stream in the test. Understood,
that such corruption is the exact reason for those tests existing.

@rishi-jat

I think the useful piece of information at this point is if the SubProcess tests
run for you, standalone, in a private sandbox environment and exhibit similar failures?

It would be nice to replicate in your sandbox environment before trying to replicate in mine.
At this point, os-lib CI seems to be consistently failing with SubProcess issues, albeit
possibly differing ones. At least we have that.

No rush at my end, just want to keep this stone rolling so that it does not collect moss.

[rishi-jat]

Thanks @LeeTibbert for the detailed pointers. I’ve split SubprocessTests.envArgs into individually named tests so CI can report exactly which case fails (e.g., envArgs.singleQuotesNoExpand). I also fixed the formatting check and pushed; CI is now rerunning.

Local results (macOS):
•JVM (Scala 2.13.16): SubprocessTests — PASS (23/23)
•Scala Native 0.5.8 (Scala 2.13.16): SubprocessTests — PASS (23/23)

No output corruption observed locally. The curl-based tests remain gated (reachability + short timeouts).

Focused reproduce commands:
./mill -i "os.jvm[2.13.16].test.testOnly" test.os.SubprocessTests
./mill -i "os.native[2.13.16].test.testOnly" test.os.SubprocessTests

If CI still fails on ubuntu-latest / JDK 11, the new test names should pinpoint the exact envArgs variant. I can iterate further or try SN 0.5.9-SNAPSHOT if helpful.

[rishi-jat]

@LeeTibbert Local runs are green any pointers on how best to debug the Ubuntu CI failures?

[LeeTibbert]

Next steps:

This is looking increasingly like a Scala Native or "SN as validly used by os-lib" problem rather
than a straight out os-lib problem. Probably no news to you.

It is looking increasingly like another incarnation of "SN sub-process under stress" failure.
Multiprocessing on SN is still fairly new (since 0.5.0) and we are still sorting out bugs,
especially under stress (both software and, as a result, developer).

In reviewing the re-worked tests, I realized that the reason that the Window SubProcess
tests do not display the problem is because they are not run. So much for my sense
of assurance in their correctness. Another problem, for another year.

Probably a good next step is for me to run some private sandbox multiprocess tests
to see if I can provoke either a sub-process failure or a broken output match, similar
to the os-lib failing tests.

A parallel effort would be to copy this PR down and exercise it on both my linux and
macOS systems, probably in a loop. Does testOnly SubProcessTest fail, either at a determinate point
or intermittently? Does SubProcessTest fail if I run the whole ensemble?

Am I correct in believing that the os-lib tests run in parallel, probably by Test class?

Problems which fail in CI but succeed on the developers system are more than
aggravating, but they prove the worth of CI.

Discussion

I was waiting for a CI run after your latest changes to SubprocessTests. Those changes are
provide, at least me, more useful information.

Drilling down to the next level, two passing CI tests for macOS are better than zero but
I doubt the underlying problem is solved there. Time will tell.

This is one failure from the linux log file, picked arbitrarily for discussion.

The filebased (and other intermittently failing) test are of the form

 test("filebased") {
      if (Unix()) {
        assert(proc(scriptFolder / "misc/echo", "HELLO").call().out.lines().mkString == "HELLO")

        val res: CommandResult =
          proc(root / "bin/bash", "-c", "echo 'Hello'$ENV_ARG").call(
            env = Map("ENV_ARG" -> "123")
          )

        assert(res.out.text().trim() == "Hello123")
      }
    }

----------------------------------- Failures -----------------------------------
2025-09-03T09:51:41.6517003Z [1948] �[31mX�[39m test.os.SubprocessTests.filebased �[2m30ms�[0m
2025-09-03T09:51:41.6517507Z [1948] �[91m�[4mos.SubprocessException�[39m�[24m: �[91mResult of /bin/bash…: 1�[39m
2025-09-03T09:51:41.6517904Z [1948] �[91mHello123�[39m
2025-09-03T09:51:41.6518267Z [1948] �[31mos.SubprocessException$.�[91mapply�[31m(�[91mUnknown�[31m)�[39m
2025-09-03T09:51:41.6518705Z [1948] �[31mos.proc.�[91mcall�[31m(�[91mUnknown�[31m)�[39m
2025-09-03T09:51:41.6519210Z [1948] �[31mtest.os.SubprocessTests$.�[91m$init$$$anonfun$1$$anonfun$8�[31m(�[91mUnknown�[31m)�[39m
2025-09-03T09:51:41.6519612Z [1948] Tests: 192, Passed: 191, Failed: 1`

Am I correct, or at least correct adjacent in interpreting the lines

2025-09-03T09:51:41.6517507Z [1948] �[91m�[4mos.SubprocessException 39m: 91 Result of /bin/bash…: 1
2025-09-03T09:51:41.6517904Z [1948] �[91mHello123�[39m

as indicating  the error likely came from 

    val res: CommandResult =
      proc(root / "bin/bash", "-c", "echo 'Hello'$ENV_ARG").call(
        env = Map("ENV_ARG" -> "123")
      )

and not (yet) the line

    assert(res.out.text().trim() == "Hello123")

I'm trying to characterize the critter we are chasing, some issue with the subprocess (probably completing and exiting
 earlier than expected) or some chunking/staccato/syncopation in the output provided by the sub-process to the 
parent.

Thanks.  I appreciate your time & effort.

[LeeTibbert]

Thank you for the change from "locally" blocks to individual "test()"
blocks. That make it much easier for at least me to figure out
in CI log files which assertions are failing. A definite improvement.

[LeeTibbert]

I think these changes will help debugging a lot. Thank you.

I'll check the logs of the next os-lib CI run and see what I can
glean.

[rishi-jat]

@LeeTibbert why CI is not running ?

[LeeTibbert]

@rishi-jat

why CI is not running ?

My reading of "This workflow requires approval from a maintainer. Learn more about approving workflows."
is that CI is waiting for approval, in the fullness of time, from @lihaoyi. I do not know who else in the os-lib
world can kick start it.

[rishi-jat]

Thanks, @LeeTibbert
@lihaoyi could you please approve the workflow so the CI can run?

[rishi-jat]

@LeeTibbert thanks for the help so far. Can you guide me on how to fix the Ubuntu CI failures so this PR can move forward?

[LeeTibbert]

@rishi-jat

Thank you for the wakeup ping.

TL; DR:

• I think this one is going to take some time; weeks, not hours.

• I am offline for most of this week, so serious work at my end will have
  to wait until I am reliably back online. Sorry for the delay.

• If you wanted, you could study to os-lib code to see if the Exception
  for a the Subprocess.bytes test below is happening because the
  apparent child exit code is 1 or because the String reply from
  the child really does mismatch (extra spaces, tabs-for-spaces, non-printable
  characters). I suggest doing so only if you have time to spare and
  nothing better to do, such as sorting your (cultural equivalent to) sock drawer.

Your thoughts?

PS: Breaking up the tests to make the failing conditions more evident really helped my debugging. Thanks!

---

I spent a long half-day tracing os-lib code & log files. Let me capture some notes.

---

I re-discovered that the:

1. Ubuntu Linux tests are being run and (obviously) failing
2. macOs tests are being run and succeeding
3. Windows tests are succeeding because the body of many of the
   tests has a guard if (Unix()) to not execute on Windows.

---

To pick one error in the logs for discussion. ``` 2025-09-10T23:21:54.3796059Z �[31mX�[39m test.os.SubprocessTests.bytes �[2m7ms�[0m 2025-09-10T23:21:54.3797605Z �[91m�[4mos.SubprocessException�[39m�[24m: �[91mResult of /home/runner/work/os-lib/os-lib/os/test/resources/test/misc/echo…: 1�[39m 2025-09-10T23:21:54.3798528Z �[91mabc�[39m ``` and the corresponding .scala code:

   test("bytes") {
      if (Unix()) {                                                           
        val res = proc(scriptFolder / "misc/echo", "abc").call()
	val listed = res.out.bytes
        listed ==> "abc\n".getBytes
      }
    }

At this point

• unclear if the Test is failing because of the child process exited with a failure code of 1,
  the 1 in the /echo…: 1 part.

• or if the listed ==> "abc\n".getBytes line is failing.
  If I strip of ANSI coloring character sequences, I get abc as the output from the child process.
  The difference is a '\n' newline, which may have been stripped off by os-lib or mistakenly by
  Scala Native. It may also be something fancy not visible to someone (me) unfamiliar with the
  os-lib, uTest framework.

That seems to be the node in the debug decision tree which will yield the most information.

---

• I tried building os-lib on my local Ubuntu machine, and mill complained that the machine
  did not have some required contemporary hardware instructions. I think I had gotten to
  that point before. I'll probably have to set up a cloud Ubuntu system.

• I did build mill on my macOS ARM system and had success run the full Test suite to on both
  JVM and SN 0.5.8. This replicates your experience and the reports from os-lib CI.
  It also helps me understand & tinker with the os-lib codebase.

[LeeTibbert]

TL;DR - I have os-lib building failing on my Ubuntu system

Progress:

I found & studied the mill-build/ docs and figured
out how to run a -jvm based mill. That worked around mill*native*.exe
not liking the hardware on my Ubuntu system. Heuristics have their limits.

That got me up and running. I was able to execute "./mill version" and get
the expected version. (I had to use a 1.0.0-RC1-jvm, 1.0.4-jvm failed).

JVM tests

To no one's surprise, but also to my elation, JVM tests passed:

• ./mill -i "os.jvm[2.13.16].test.testOnly" test.os.SubprocessTests
• ./debug-subprocess-loop.sh passed all 50 sequential executions

This establishes a baseline in my environment.

Scala Native 0.5.8 tests

• The initial run generated so much smoke I had to call the county fire department
  and ask them not to turn out.

• Various runs after would usually have 1 to 4-ish errors after that,
  all of the familiar Exception where the exit code was 1.

• I altered SubprocessTests.stressSubprocess to do 10_000 sequential
  iterations (was). Various executions of that all succeeded.

  This adds evidence to a hypothesis that at least one of the underlying defects is due to
  a number of sub-processes executing at once and changing the timing of, probably,
  exitValue() calls.

Next steps

As previously mentioned, this week will be short for me. I may not get much debugging done.
When I do have cycles:

• I need to trace why os-lib is throwing an Exception when the process exit value is not zero.
  It looks like the child process has returned the expected text/bytes. exitValue is not really
  a concern, or rather, possibly a later concern.

• I need to insert a test for exitValue after the sub-process call(). I need to learn if
  os-lib does a waitfor on the process or if I need to do that myself, before the exitValue.
  Does execution reach this point, or is the Exception in the call() itself?

• If the exitValue assert passes, is the count of characters in the response from the
  child the same as expected? This helps rule out non-printing characters in the response.

Updates

• 2025-09-16:30 UTC
  Skipping over a lot of debugging and focusing on the salient points.

  • The exitValue check and resultant os.SubprocessException are part of the definition of
    os-lib call() in os/src/ProcessOps.scala.

  • When I change the .call() sites in SubprocessTests to use check = false, a
    copy of debug_subprocess-loop modified to use Scala Native succeeds 45 times out of 50 (one run).

  • An failure that I have seen before but which is easier to see in the debug_SN_subprocess-loop
    above is:

java.io.IOException: pidfd_open failed: No such process
    java.lang.process.UnixProcessGen2.linuxWaitForImpl(Unknown)
    java.lang.process.UnixProcessGen2.osWaitForImpl(Unknown)
    java.lang.process.UnixProcessGen2.waitFor(Unknown)
    os.SubProcess.waitFor(Unknown)
    os.ProcessLike.join(Unknown)
    os.SubProcess.join(Unknown)
    os.proc.call(Unknown)

 That is a 'should never happen'. Looks like an attempt to `join()` an operating system
 process which is long since gone (or never existed).

 Whatever the cause, that situation is not good.

• Over the next sessions, I need to drill down on the check = true path in
  os/src/ProcessOps.scala and understand its logic. Is the underlying
  SN javalib Process truly returning an exit code of 1, or is something else
  happening so that a layer of os-lib interprets as an exit code of 1?

  • The usual practice of swapping in a debug SN 0.5.8 or 0.5.9-SNAPSHOT
    to see what the child thinks it is returning is not a trivial exercise.

[LeeTibbert]

Quick status: 2025-09-17 14:45 UTC ish

I've come up with some Scala Native 0.5.8 specific changes to SubprocessTests which
allow all but one test to run in a meaningful, useful way.

To pick one run. There were 16 failures in a run of 8_000 iterations. That gives
an overall failure rate of 0.2%. Not perfect, but better than it was. The failures
were SN exceptions all on the same test.

I believe I know a change that I can make to Scala Native to make that one test pass.
I do not know when the SN 0.5.9 bus is leaving and I do not know if I will be able
to make this change and test it before then.

The fix I am thinking about makes, I hope, the symptom go away. There may
be deeper defects that will have to be solved in further iterations. I think 'fragile'
is the word I am searching for.

As mentioned, I am out for the rest of the week. I'll post my recommend changes
when I get back.

os-lib is doing a pretty good job of exercising paths in SN.

[LeeTibbert]

@rishi-jat

Status: 2025-09-26 13:30ish UTC

TL;DR - This PR is going to need SN 0.5.9 at the least

• I do not know what the os-lib workflow is. In SN I would be converting this PR to Draft
  because it is going to be around for awhile.

• I've been working pretty intensively over the past period on
  getting these os-lib to run consistently on Linux. I have not
  tried macOS yet.

• Another SN contributor merged a Process related PR into
  SN 0.5.9-SNAPSHOT.

• I've done many runs of SubprocessTests both with with my eventually superseded changes and
  the eventually merged PR. These used your 'loop' script and local scripts based on that.
  Your script saved a lot of time. I have sent mental praise many a time for both the arguments
  and the way you documented them in even a rough script. Saves me editing time & confusion.
  These days, how does one say "good craftsmanship" in a gender neutral and/or non-offensive way?

  At this point, I can reliably do somewhere between 300 to 600 iterations of the loop before I get
  a hang. I check my operating system info and that of the processes involved and it is a true
  zero-cpu hang. (I say that because at one point I was getting 100% CPU busy wait loops
  because isAlive() was always returning true. ) One process, probably the utest framework,
  is waiting in network recvfrom(). The other, probably the running tests, appears to be
  waiting for a Future to complete. The uncompleted Future is probably notification of a child exit.
  I have not yet gotten to the point of proving that to myself.

  Work continues.

  All this means that the current "nightly" 0.5.9-SNAPSHOT solves a lot of problems but
  probably not all.

  You can try the nightly yourself if you want to or you could wait until I report that the stress test
  above is fixed. That should pass "one shot" tests but probably not stress tests (unless your
  environment is more provident than mine).

  Changing "0.5.8" or such in this PR to "0.5.9-SNAPSHOT" and using the same resolver may work. I have
  not tried that.

[rishi-jat]

Hi @LeeTibbert,

Thanks a lot for the detailed update! I’m also working locally to reproduce and debug the Ubuntu CI failures. I’ll keep iterating and push any changes as soon as I have something useful. Hopefully, we can make this PR stable on Linux soon.

Appreciate all your guidance so far!

[LeeTibbert]

@rishi-jat

Two questions, if you please. These are meant as a 5 minute each "do you know off the top of your
head" questions, not a week long time suck. Thank you for any help, including "Beats me, let me know
when you find out."

1. I noticed that the os-lib tests are always run in the default SN "debug" mode. Because I am
   doing hundreds and thousands of 'loop' iterations and monitoring/watching in real time, I'd like
   to use Scala Native's "release-fast" mode. In later cycles, I might add "LTO" (long term optimization).
   Payoff of the first is that it reduces my wall clock time, I hope. The advantage of being able
   to exercise the tests with sensible combinations of the three is that sometimes each of the
   latter two show off bugs not seen in "debug" mode.

   I looked in the build.mill and, limited by my lack of mill knowledge & experience, did not
   see any obvious way.

2. It appears that os-lib is supported on Windows. Is SubprocesTest expected to run
   on Windows (Windows 10, 11, or both) where the sections marked if (Unix()) are skipped?

   When/If I figure out the 300-500 hang, I'd eventually like to try os-lib tests using SN 0.5.n
   on Windows. Just to exercise that path before somebody tries to use it and discovers it
   "never tested" broken.

[rishi-jat]

Hi @LeeTibbert,

Thanks for the detailed analysis! I've added comprehensive debugging to help us figure out what's going wrong on Ubuntu.

What I've enhanced:

• All the failing subprocess tests now show detailed error info - whether it's the subprocess itself crashing (exit code != 0) or if it's an output mismatch
• Added specific debugging for bytes, envWithValue, workingDirectory, and the destroy/destroyNoGrace tests that keep failing
• Bumped the retry count on destroyNoGrace from 3 to 5 since it seems flaky
• Every subprocess call now reports exit codes, stderr, and exact output differences when things go wrong

Local testing:

• JVM: All 24 SubprocessTests pass consistently
• Scala Native 0.5.8: All 24 tests pass as well
• The stress test runs fine locally (no corruption seen)

The enhanced error messages should tell us exactly what you were asking about - is it the subprocess failing to run properly, or is it completing but with corrupted/wrong output?

CI is running now with the new diagnostics. Hopefully this gives us the smoking gun we need to track down why Ubuntu is different from macOS/Windows.

Let me know what the logs show!

This sounds more conversational and natural while covering all the important technical points.

[rishi-jat]

@rishi-jat

Two questions, if you please. These are meant as a 5 minute each "do you know off the top of your head" questions, not a week long time suck. Thank you for any help, including "Beats me, let me know when you find out."

1. I noticed that the os-lib tests are always run in the default SN "debug" mode. Because I am
   doing hundreds and thousands of 'loop' iterations and monitoring/watching in real time, I'd like
   to use Scala Native's "release-fast" mode. In later cycles, I might add "LTO" (long term optimization).
   Payoff of the first is that it reduces my wall clock time, I hope. The advantage of being able
   to exercise the tests with sensible combinations of the three is that sometimes each of the
   latter two show off bugs not seen in "debug" mode.
   I looked in the build.mill and, limited by my lack of mill knowledge & experience, did not
   see any obvious way.
2. It appears that os-lib is supported on Windows. Is SubprocesTest expected to run
   on Windows (Windows 10, 11, or both) where the sections marked if (Unix()) are skipped?
   When/If I figure out the 300-500 hang, I'd eventually like to try os-lib tests using SN 0.5.n
   on Windows. Just to exercise that path before somebody tries to use it and discovers it
   "never tested" broken.

@LeeTibbert

1. Scala Native release modes in Mill: You can override the release mode by adding this to the OsNativeModule in
   /Users/rishijat/Documents/os-lib/build.mill :

def nativeMode = mill.scalanativelib.api.ReleaseMode.ReleaseFast
// or ReleaseMode.ReleaseFull for LTO

Currently it defaults to Debug mode. For your stress testing, ReleaseFast should definitely help with wall clock time. You could also add a system property to toggle it:

def nativeMode = sys.props.get("native.mode") match {
  case Some("release-fast") => mill.scalanativelib.api.ReleaseMode.ReleaseFast  
  case Some("release-full") => mill.scalanativelib.api.ReleaseMode.ReleaseFull
  case _ => mill.scalanativelib.api.ReleaseMode.Debug
}

Then run with: : ./mill -Dnative.mode=release-fast os.native[2.13.16].test

2. SubprocessTests on Windows: Yes, SubprocessTests should run on Windows but most tests are skipped due to the if (Unix()) guards. Looking at the CI logs, Windows tests do run and pass (19s execution time), but they're essentially testing very little of the subprocess functionality.

The few tests that do run on Windows are the non-Unix ones like listMixAndMatch (which has Windows-specific quote handling) and some basic path/string operations.

You're right that this is a gap - the subprocess functionality is largely untested on Windows. If you do get a chance to test SN 0.5.n on Windows, that would definitely help catch "never tested" issues before users hit them.

Let me know if you need help modifying the build file for the release modes!

[LeeTibbert]

rishi-jat

Thank you for all the improvements and the info about SN build modes
and also Windows.

Then run with: : ./mill -Dnative.mode=release-fast os.native[2.13.16].test

Ah! a concrete example helps my frazzled mind.

Windows

I had hoped to update a machine to Windows 11 but that effort failed. Argh!
So Windows has receded to being a problem for another year or century.

    JVM: All 24 SubprocessTests pass consistently
    Scala Native 0.5.8: All 24 tests pass as well
    The stress test runs fine locally (no corruption seen)

I am currently in the process of establishing new baselines using the
most current SN 0.5.9-SNAPSHOT. I am constantly reminded about
the Greek philosopher who said that no one ever steps into the same
river twice. Change is pretty fast and hard to keep up.

When I get stable there once again, I will have to try the new Tests.
The fact that they pass on Scala Native 0.5.8 is paradoxical from
my end. That version was pretty broken, at least when it came to
the intermittent error code of 1 (general error).

Let me get settled in.

The SN 'nightly' versioning scheme has changed in the past
few days. When I some repeatable & believable results using
a defined SN 0.5.9-'nightly', I can describe you can try it.
Running out the door now, sorry to be terse.

[LeeTibbert]

rishi-jat

I tried to implement your suggestion about nativeMode by making the edit below.
I still seem to be getting SN debug mode builds.

The def scalaNativeVersion = "0.5.9-SNAPSHOT" appears to be working just fine
(i.e, blows up when I do not have that version in my ~.ivy2/local cache and works
when I do.)

I am using the JVM mill (to avoid minimal required hardware issues), so that might be a
bit different.

Thanks. This is not a biggie, but a 'nice to have'.

 object native extends Cross[OsNativeModule](scalaVersions)
  trait OsNativeModule extends OsModule with ScalaNativeModule {
//    def scalaNativeVersion = "0.5.8"

    def scalaNativeVersion = "0.5.9-SNAPSHOT"
    def nativeMode = mill.scalanativelib.api.ReleaseMode.ReleaseFast

    object test extends ScalaNativeTests with OsLibTestModule {

I am still chasing intermittent hangs (two or more processes waiting on each other, zero CPU).
I suspect that it has something to do with os-lib 'spawn()', hence 'call()' using 'destroy()' and
'destroyForcibly()', especially the latter, by default. Timing issues of when a SIGKILL is received.
Fun, Fun, Fun

[LeeTibbert]

@rishi-jat

Status: 2025-10-06 09:00 UTC

Some success!

Everything below uses a private copy of 0.5.9-SNAPSHOT (more about SNAPSHOT) in
a separate entry.

A 4000 iteration run, and many smaller runs smaller than that, all succeeded when
I ran an experiment which updated many layers of the software and changed the SN
compilation mode.

I have another 4000 run currently executing which falls back to SN mode=debug to
factor that out.

4000 was selected as large-ish number which fits into a rest period of many hours, not a full day.

I used the latest tests from this PR, but have not yet had a chance to examine them
line by line. Haven't need to. Sorry, will get to that now that I seem to be making some
progress.

The software upgrades:

1. Scala 3.3.6

2. Mill JVM 1.0.6

3. os-lib 0.11.5, with most recent SubProcessTests from this PR

4. Scala Native 0.5.9-SNAPSHOT built with mode=release-fast & lto=thin

I am continuing work on resolving this PR.

As your time allows, please let me know your thoughts.

---

In case you are interested I isolated the line that was provoking Scala Native to ProcessOps.scala:

  // FIXME - 2025-10-05 08:54 -0400 Try bypassing 1 millisecond isAlive pounding                                                                      
    //          while (proc.wrapped.isAlive) Thread.sleep(1)                                                                                            

          while (proc.wrapped.isAlive)
            Thread.sleep(1000 * 1)

This was a sandbox change, the runs described above did not contain this.

With the change to 1 full second, a 4000+ iteration run not described above
completed successfully, using Scala 2.12.mumble.

Later, 2025-10-09 10:40 UTC: In multiple additional runs, I was unable to
reproduce this workaround. Sic transit.

Let me see what the results of my release-mode=debug run are early
this evening, my time before making any suggestions about that
sleep timing on SN. I am well aware that there is a clock running on
this PR.

---

> Scala Native 0.5.9-SNAPSHOT built with mode=release-fast & lto=thin

I used a SN feature which is no longer documented to set the 'mode' and 'lto'.
'lto' is "Link Time Optimization".

I was focused on solving the main "intermittent hang" problem and backed off
trying to set these in build.mill as you described above after a few failures.

[LeeTibbert]

@rishi-jat

Status: 2025-10-06 09:00 UTC

I've spent many an hour on this Issue since the last update and tried a number of things
and run thousands of repetitions.

There is talk in the SN world about an SN 0.5.9 happening Real Soon Now (weeks).
Given that, I suggest:

1. Wait until SN 0.5.9 is released, update this PR and try again.

2. Add an os-lib release note stating that on Linux occasional process zero
   cpu non-terminating waits (a.k.a hangs) have been seen when a moderate number
   of os-lib call() or spawn() are made within a very short period. 'moderate'
   is defined as the range from tens to 1500. "short" is defined as the time period
   which reveals the defect.

   This issue is under active investigation by Scala Native development.

   There are three know ways to avoid this defect:

   1. build the application (how?) with Scala Native release mode release-fast AND lto lto=thin.
      The former by itself has not worked for the Scala Native developer.

      The workaround has been exercised using only 'most recent' versions of Scala, Mill
      and associated software.

      Many applications may use these Scala Native settings for release but not for testing.
      Using them for both may provide relief if this defect is seen in the latter.

   2. Provide the os-lib argument destroyOnExit = false in all calls to os-lib call() and spawn() .

      This workaround has been exercised on Scala 2.12, so is less sensitive to software versions.

   3. Do both.

---

FYI (For your information):

• I ran my usual 4000 iteration run on macOS with days old SN 0.5.9-SNAPSHOT and encountered
  no problem.

• On Linux, I tried increasing the 'watcher' Thread.sleep() to 2 seconds. Still encountered the problem.

• On Linux, I tried using LockSupport.parkNanos() with various times, up to 2 seconds and still
  encountered the problem. At least on SN parkNanos() exercises a much simpler, and probably
  faster code path.

[rishi-jat]

Hi @LeeTibbert,

Thank you so much for the detailed updates and for all the time you’ve spent debugging this PR. I really appreciate your thorough analysis and the experiments you’ve done with Scala Native 0.5.9-SNAPSHOT.

Noted on the current status:

• I understand that the main intermittent hang issue on Linux is still under active investigation by the Scala Native team.

• Waiting for the official SN 0.5.9 release before making further updates to this PR is the right approach.

• The suggested workarounds—using release-fast + lto=thin and destroyOnExit = false—make sense and will be documented in the os-lib release notes as you recommended.

I’ll hold off on any further changes until SN 0.5.9 is officially released. Once it is out, I can update this PR accordingly and test again to ensure stability.

Thanks again for all your guidance and for validating the local improvements it’s been extremely helpful.

[LeeTibbert]

@rishi-jat

Thank you for the timely update. Sounds like a plan. Also sounds like
the collegial environment focused on the 'common good' that Open Source Software
should be.

Merit to you for your patience. os-lib and you have certainly helped Scala Native
by getting some bugs to reveal themselves, especially the "hang at moderate scale".

Since we are in a Future.await() state, feel free to ping me if this Issue itself appears 'hung'.

L.

[LeeTibbert]

@rishi-jat

Scala Native 0.5.9 was released minutes ago. When you get some time, you
could try this PR using the released 0.5.9 and see what smokes.

I am offline until Wednesday, UTC, or so, but my hopes and best wishes will be with you.