broken support for use of wildcards in the compiler classpath on Windows

[philwalk]

Minimized code

java -Xmx20480m -Dnet.ipv6.bindv6only=0 -Dfile.encoding=UTF-8 -classpath "%CP%" -Dscala.usejavacp=true dotty.tools.dotc.Main -classpath 'c:/opt/uejlib2.13/*'  c:/opt/ue/jsrc/s3.sc

Output (click arrow to expand)

Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <*> at index 18: c:/opt/uejlib2.13/*
        at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
        at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
        at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
        at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92)
        at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:229)
        at java.base/java.nio.file.Path.of(Path.java:147)
        at java.base/java.nio.file.Paths.get(Paths.java:69)
        at dotty.tools.dotc.core.MacroClassLoader$.$anonfun$1(MacroClassLoader.scala:23)
        at scala.collection.ArrayOps$.map$extension(ArrayOps.scala:924)
        at dotty.tools.dotc.core.MacroClassLoader$.makeMacroClassLoader(MacroClassLoader.scala:23)
        at dotty.tools.dotc.core.MacroClassLoader$.init(MacroClassLoader.scala:20)
        at dotty.tools.dotc.Driver.setup(Driver.scala:72)
        at dotty.tools.dotc.Driver.process(Driver.scala:192)
        at dotty.tools.dotc.Driver.process(Driver.scala:162)
        at dotty.tools.dotc.Driver.process(Driver.scala:174)
        at dotty.tools.dotc.Driver.main(Driver.scala:201)
        at dotty.tools.dotc.Main.main(Main.scala)

Use of wildcards in the classpath has been supported by java since java 6. See stack overflow: including-all-the-jars-in-a-directory-within-the-java-classpath

It has also been supported at least since scala 2.8, and is critical for working around OS command line length limitations.

The current implementation assumes that every classpath entry is a file, leading to a compiler crash when the glob entry is encountered.

An alternative way to avoid long classpath problems has been available since java 9+, and is described here (see @raman's entry):

stackoverflow: using an argument file to pass classpath entries

Briefly, it the classpath is specified in a text file (e.g., cparg), which is passed to java like so:

java @c:\path\to\cparg

A similar approach could be implemented in dotty.tools.dotc.Main.main, although ideally it would allow an os-independent format, not requiring the use of path separators (e.g., one classpath entry per line).

[smarter]

scalac @foo already exists in Scala 2 and 3 and will read any argument from the file foo

[smarter]

I've just tried it on linux and scalac -classpath 'foo/*' test.scala worked as expected, so I'm guessing this is specific to Windows? @liufengyun do you think you could investigate?

[smarter]

FWIW, the code that deals with * in classpath is https://github.com/lampepfl/dotty/blob/dbc11869127dd2e5cec205b7d64e58cf5488d2a6/compiler/src/dotty/tools/io/ClassPath.scala#L134-L142, but the crash happens in MacroClassLoader which directly passes the paths from ctx.settings.classpath to java.nio.file.Paths.get without preprocessing the *: https://github.com/lampepfl/dotty/blob/dbc11869127dd2e5cec205b7d64e58cf5488d2a6/compiler/src/dotty/tools/dotc/core/MacroClassLoader.scala#L23

[philwalk]

The problem does seem to be specific to Windows, no such problem in Linux. I will investigate further this afternoon.
By the way, in Linux I was able to do this:

scalac -classpath 'c:/opt/uejlib2.13/*' /opt/ue/jsrc/s3.sc

The invocation in Windows that I provided above is basically the java command that resulted when I issued the same command from a cygwin session, so it's possible that the "scalac" bash script behaves incorrectly in Windows. Is there a plan to provide Windows batch files? To be clear, I'm not advocating the need for scalac.bat, etc., but am curious.

[smarter]

I don't think the script is the source of the problem, the problem is we pass wildcards to java.nio.file.Paths.get which happens to work on Linux but not Windows.

[philwalk]

When I reduce the code in dotty/compiler/src/dotty/tools/dotc/core/MacroClassLoader.scala line 23, to the following script:

val cp = "/opt/uejlib2.13/*"
val url = java.nio.file.Paths.get(cp).toUri.toURL
println(url)

I get different behavior in Windows versus Linux:

Windows:

java.nio.file.InvalidPathException: Illegal char <*> at index 16: /opt/uejlib2.13/*
        at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
        at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
        at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
        at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92)
        at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:229)
        at java.base/java.nio.file.Path.of(Path.java:147)
        at java.base/java.nio.file.Paths.get(Paths.java:69)
        at Main$$anon$1.<init>(starTest.sc:4)
        at Main$.main(starTest.sc:2)
        at Main.main(starTest.sc)
        [snip remaining stack dump]

Linux:

file:/opt/uejlib2.13/*

So java.nio.file.Path treats a wildcard entry differently in Windows.

This seems to explain why:

https://stackoverflow.com/questions/27522581/asterisks-in-java-path

The short version: asterisk is a legal filename character in Linux, but not in Windows.

I will investigate using the scalac @foo approach.

[philwalk]

FYI, the compiler doesn't currenty permit source files to be on a drive other than the current session drive.

My /tmp directory is mounted to G: (a disk drive) to avoid wearing out my SSD drive.
When I try to compile a temporary copy of my scala script (with the hash-bang line removed), I get this:

/opt/scala3/bin/scalac @.uejlib213Classpath "G:/tmp/s3.sc"

Output (click arrow to expand)

exception occurred while compiling G:\tmp\s3.sc

java.lang.IllegalArgumentException: 'other' has different root while compiling G:/tmp/s3.sc
Exception in thread "main" java.lang.IllegalArgumentException: 'other' has different root
        at java.base/sun.nio.fs.WindowsPath.relativize(WindowsPath.java:404)
        at java.base/sun.nio.fs.WindowsPath.relativize(WindowsPath.java:42)
        at dotty.tools.dotc.transform.PostTyper$PostTyperTransformer.transform(PostTyper.scala:346)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform$$anonfun$2(Trees.scala:1369)
        at scala.collection.immutable.List.mapConserve(List.scala:472)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1369)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transformStats(Trees.scala:1367)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1354)
        at dotty.tools.dotc.transform.MacroTransform$Transformer.transform(MacroTransform.scala:54)
        at dotty.tools.dotc.transform.PostTyper$PostTyperTransformer.transform(PostTyper.scala:417)
        at dotty.tools.dotc.transform.MacroTransform.run(MacroTransform.scala:21)
        at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:296)
        at scala.collection.immutable.List.map(List.scala:246)
        at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:297)
        at dotty.tools.dotc.Run.runPhases$4$$anonfun$4(Run.scala:185)
        at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
        at dotty.tools.dotc.Run.runPhases$5(Run.scala:195)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:203)
        at dotty.runtime.function.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:67)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:210)
        at dotty.tools.dotc.Run.compileSources(Run.scala:147)
        at dotty.tools.dotc.Run.compile(Run.scala:129)
        at dotty.tools.dotc.Driver.doCompile(Driver.scala:38)
        at dotty.tools.dotc.Driver.process(Driver.scala:193)
        at dotty.tools.dotc.Driver.process(Driver.scala:162)
        at dotty.tools.dotc.Driver.process(Driver.scala:174)
        at dotty.tools.dotc.Driver.main(Driver.scala:201)
        at dotty.tools.dotc.Main.main(Main.scala)
compile error

A workaround is to pushd g:/tmp, compile, and then popd.

[liufengyun]

@philwalk This is a bug (#10763). Before we fix that, a workaround is to pass the option -sourceroot G:/tmp.

[philwalk]

@liufengyun wow, good to know!

[michelou]

@philwalk You may also be interested in my GitHub repo dotty-examples about running Scala 3 on Windows and - in relation with this issue - I invite you to check lines 470 and 502 of the batch file enumPlanet/build.bat.

PS. People working with Cygwin Bash or MSYS2 Bash can use the shell script enumPlanet/build.sh

[philwalk]

@michelou Thanks, that looks like an interesting resource.

[philwalk]

A correction to my previous comment:

It seems that you can specify the classpath EITHER via -classpath or @argsfile, but not both.

Perhaps the one that appears later on the command line (? just a guess) supercedes the previous one.
So a workable solution to the very-long-classpath problem is to dynamically generate an @argsfile.

In a scripting environment, it would be nice to be able to specify a static @argsfile, but it doesn't appear to be feasible.

[philwalk]

This seems to be partially fixed in the current compiler:
Scala compiler version 3.0.0-RC2-bin-SNAPSHOT-git-58c77c3 -- Copyright 2002-2021, LAMP/EPFL

Here's my test script, named processTest.sc:

#!./bin/scala -classpath './ulib/*'
!#

// compile failure unless these imports are resolved via above -classpath
import better.files._
import better.files.Dsl._

@main def m() =
  printf("Testing wildcard classpath entries below '%s'.\n",".")
  printf("Testing wildcard classpath entries below '%s'.\n",cwd)

I prep my test environment like this:

mkdir -p lib
ln -sFT $BETTER_FILES_PATH lib/better-files.jar
 ./processTest.sc

In my Linux environment, the script compiles (imports are resolved), but there's a runtime exception:

$ touch ./processTestLinux.sc ;  ./processTestLinux.sc
Testing wildcard classpath entries below '.'.
Exception in thread "main" java.lang.NoClassDefFoundError: better/files/Dsl$
        at processTestLinux$package$.m(processTestLinux.sc:10)
        at m.main(processTestLinux.sc:8)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:39)
        at dotty.tools.scripting.Main$.main(Main.scala:39)
        at dotty.tools.scripting.Main.main(Main.scala)
Caused by: java.lang.ClassNotFoundException: better.files.Dsl$
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 9 more

There is other evidence, but I'll have to get back later.

[michelou]

@philwalk I assume I should read

• './lib/*' instead of ./ulib/*' and
• ./processTest.sc instead of ./processTestLinux.sc
  in your session examples, right ?!

[philwalk]

@michelou - yes, your corrections are valid, I updated my comment.

I have a fix for this issue and can create a PR soon, if tests look good.

The problem has a couple of facets:

1. with the -save option, the jar file manifest Class-Path requires wildcard entries to be expanded
2. the non-Windows runtime also requires classpath entries to be expanded

The wildcard entries are already expanded in Windows when I examine them at dotty/compiler/src/dotty/tools/dotc/core/MacroClassLoader.scala line 23.

That solves the original crash symptom:

Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <*> at index 18: c:/opt/uejlib2.13/*

update on 2021-03-11: does not actually solve the original crash problem for all use-cases, see #11633 comment at line 71 of dist/bin/scala

The PR should solve the remaining runtime classpath problems.