scala_legacy crash caused by any jar in classpath with empty Class-Path: attribute

[philwalk]

Compiler version

# scala -version
Scala code runner version: 1.5.4
Scala version (default): 3.6.3

Minimized code

#!/usr/bin/env -S scala_legacy -classpath ./pallet_3.jar

object LegacyBug {
  def main(args: Array[String]): Unit = {
    printf("hello legacy!\n")
  }
}

Can be reproduced by a jar with empty Class-Path: attribute.

If the jar has an empty Class-Path: attribute in the MANIFEST, it crashes.
If it has no Class-Path: attribute, or a non-empty Class-Path: attribute, no problem.

Output

[warning] MainGenericRunner class is deprecated since Scala 3.5.0, and Scala CLI features will not work.
[warning] Please be sure to update to the Scala CLI launcher to use the new features.
[warning] Check the Scala 3.5.0 release notes to troubleshoot your installation.

  Exception while compiling C:\opt\ue\jsrc\legacyBug.sc

  An unhandled exception was thrown in the compiler.
  Please file a crash report here:
  https://github.com/scala/scala3/issues/new/choose
  For non-enriched exceptions, compile with -Xno-enrich-error-messages.


     while compiling: <no file>
        during phase: <some phase>
                mode: Mode()
     library version: version 2.13.15
    compiler version: version 3.6.3
            settings: -classpath C:/opt/scala3/lib/scala.jar;C:/opt/scala3/lib/with_compiler.jar;emptyClassPath.jar -d C:\tmp\scala3-scripting13458733512999800816

Exception in thread "main" java.lang.NullPointerException: Cannot invoke "dotty.tools.io.AbstractFile.ext()" because "file" is null
	at dotty.tools.dotc.classpath.FileUtils$.isJarOrZip(FileUtils.scala:37)
	at dotty.tools.dotc.classpath.ClassPathFactory$.newClassPath(ClassPathFactory.scala:94)
	at dotty.tools.dotc.classpath.ClassPathFactory.newClassPath(ClassPathFactory.scala:20)
	at dotty.tools.dotc.classpath.ClassPathFactory.$anonfun$2$$anonfun$3(ClassPathFactory.scala:73)
	at scala.collection.Iterator$$anon$9.next(Iterator.scala:584)
	at scala.collection.immutable.List.prependedAll(List.scala:153)
	at scala.collection.immutable.List$.from(List.scala:685)
	at scala.collection.immutable.List$.from(List.scala:682)
	at scala.collection.IterableOps$WithFilter.map(Iterable.scala:900)
	at dotty.tools.dotc.classpath.ClassPathFactory.$anonfun$2(ClassPathFactory.scala:69)
	at scala.collection.immutable.List.flatMap(List.scala:294)
	at dotty.tools.dotc.classpath.ClassPathFactory.classesInPathImpl(ClassPathFactory.scala:68)
	at dotty.tools.dotc.classpath.ClassPathFactory.classesInExpandedPath(ClassPathFactory.scala:46)
	at dotty.tools.dotc.config.PathResolver$Calculated$.basis(PathResolver.scala:232)
	at dotty.tools.dotc.config.PathResolver$Calculated$.containers$lzyINIT1(PathResolver.scala:236)
	at dotty.tools.dotc.config.PathResolver$Calculated$.containers(PathResolver.scala:236)
	at dotty.tools.dotc.config.PathResolver.containers(PathResolver.scala:258)
	at dotty.tools.dotc.config.PathResolver.result$lzyINIT1(PathResolver.scala:261)
	at dotty.tools.dotc.config.PathResolver.result(PathResolver.scala:260)
	at dotty.tools.dotc.config.JavaPlatform.classPath(JavaPlatform.scala:18)
	at dotty.tools.dotc.config.JavaPlatform.rootLoader(JavaPlatform.scala:38)
	at dotty.tools.dotc.core.Contexts$ContextBase.rootLoader(Contexts.scala:909)
	at dotty.tools.dotc.core.Definitions.RootClass$$anonfun$1(Definitions.scala:203)
	at dotty.tools.dotc.core.Symbols$.$anonfun$2(Symbols.scala:655)
	at dotty.tools.dotc.core.Symbols$.newClassSymbol(Symbols.scala:589)
	at dotty.tools.dotc.core.Symbols$.newModuleSymbol(Symbols.scala:655)
	at dotty.tools.dotc.core.Symbols$.newPackageSymbol(Symbols.scala:716)
	at dotty.tools.dotc.core.Definitions.RootClass(Definitions.scala:203)
	at dotty.tools.dotc.core.Denotations$.recurSimple$1(Denotations.scala:1352)
	at dotty.tools.dotc.core.Denotations$.recur$1(Denotations.scala:1356)
	at dotty.tools.dotc.core.Denotations$.staticRef(Denotations.scala:1360)
	at dotty.tools.dotc.core.Symbols$.requiredPackage(Symbols.scala:944)
	at dotty.tools.dotc.core.Definitions.ScalaPackageVal(Definitions.scala:215)
	at dotty.tools.dotc.core.Definitions.ScalaPackageClass(Definitions.scala:218)
	at dotty.tools.dotc.core.Definitions.AnyClass(Definitions.scala:281)
	at dotty.tools.dotc.core.Definitions.syntheticScalaClasses(Definitions.scala:2184)
	at dotty.tools.dotc.core.Definitions.syntheticCoreClasses(Definitions.scala:2199)
	at dotty.tools.dotc.core.Definitions.init(Definitions.scala:2215)
	at dotty.tools.dotc.core.Contexts$ContextBase.initialize(Contexts.scala:922)
	at dotty.tools.dotc.core.Contexts$Context.initialize(Contexts.scala:544)
	at dotty.tools.dotc.Run.rootContext(Run.scala:502)
	at dotty.tools.dotc.Run.<init>(Run.scala:523)
	at dotty.tools.dotc.Compiler.newRun(Compiler.scala:178)
	at dotty.tools.dotc.Driver.doCompile(Driver.scala:35)
	at dotty.tools.scripting.ScriptingDriver.compileAndRun(ScriptingDriver.scala:22)
	at dotty.tools.scripting.Main$.process(Main.scala:39)
	at dotty.tools.MainGenericRunner$.run$1(MainGenericRunner.scala:250)
	at dotty.tools.MainGenericRunner$.process(MainGenericRunner.scala:286)
	at dotty.tools.MainGenericRunner$.main(MainGenericRunner.scala:297)
	at dotty.tools.MainGenericRunner.main(MainGenericRunner.scala)

Expectation

Should not crash.
The following script verifies that scala3-3.3.1 has no problem with the jar file.

#!/opt/scala3-3.3.1/bin/scala -classpath ./emptyClassPath.jar
object LegacyBug {
  def main(args: Array[String]): Unit = {
    printf("hello legacy!\n")
  }
}

An empty string is a legal classpath entry, interpreted by the jvm as the current working directory, although it might have been created by accident. The following entry in build.sbt can produce a problem jar:

Compile / packageBin / packageOptions += Package.ManifestAttributes(java.util.jar.Attributes.Name.CLASS_PATH -> "")

Here's a script to create a problem jar named emptyClassPath.jar in the current directory:

#!/bin/bash

HERE=`pwd -P`
WORK_NAME=emptyClassPath
PROBLEM_JAR="$HERE"/$WORK_NAME.jar

WORK_DIR=/tmp/${WORK_NAME}$$
MANIFEST="$WORK_DIR"/META-INF/MANIFEST.MF
set -x
mkdir -p $WORK_DIR/META-INF

cd $WORK_DIR
touch main.class
cat > $MANIFEST <<- EOF
Manifest-Version: 1.0
Class-Path:  
Created-By: $WORK_NAME
EOF
cat $MANIFEST
/opt/jdk8/bin/jar -cmf $MANIFEST "$PROBLEM_JAR" .

[som-snytt]

TIL scala_legacy. https://scala-cli.virtuslab.org/docs/guides/introduction/old-runner-migration/

➜  snips ~/projects/dotty/bin/scalaQ --classpath badcp.jar hello-import.scala
[warning] MainGenericRunner class is deprecated since Scala 3.5.0, and Scala CLI features will not work.
[warning] Please be sure to update to the Scala CLI launcher to use the new features.
[warning] Check the Scala 3.5.0 release notes to troubleshoot your installation.
hello, world
➜  snips jar tf badcp.jar
META-INF/
META-INF/MANIFEST.MF
➜  snips jar xf badcp.jar
➜  snips cat META-INF/MANIFEST.MF
Manifest-Version: 1.0
Class-Path:
Created-By: 23.0.2 (Eclipse Adoptium)

➜  snips scala -version
Scala code runner version: 1.5.4
Scala version (default): 3.6.2
➜  snips scala --classpath badcp.jar hello-import.scala
Downloading compilation server 2.0.5
Starting compilation server
Compiling project (Scala 3.6.2, JVM (23))
Compiled project (Scala 3.6.2, JVM (23))
hello, world

I haven't tried shebang yet. I always resist starting a compilation server. It took so many years to deprecate fsc.

➜  snips scala-cli run --server=false --jvm 23 -S scala_legacy --classpath badcp.jar i22459.scala
[error]  Cannot find matching Scala version for 'scala_legacy'
You can only choose one of the 3.x, 2.13.x, and 2.12.x. versions.
The latest supported stable versions are 2.12.20, 2.13.15, 3.5.2.
In addition, you can request compilation with the last nightly versions of Scala,
by passing the 2.nightly, 2.12.nightly, 2.13.nightly, or 3.nightly arguments.
Specific Scala 2 or Scala 3 nightly versions are also accepted.
You can also request the latest Scala 3 LTS by passing lts or 3.lts.

Worth noting that an empty attribute is not per spec:

The manifest for an application can specify one or more relative URLs referring to the JAR files and directories for other libraries that it needs.

https://docs.oracle.com/en/java/javase/23/docs/specs/jar/jar.html#class-path-attribute

[philwalk]

@snips - thanks for pointing out scalaQ, that's useful. It might provide the elusive migration path I've been needing (I've been unable to migrate beyond 3.4.3 for a few months now.)

[philwalk]

@som-snytt
The following variation on your scala-cli command line doesn't encounter errors, but the main method never gets called, due to the ".sc" extension:

scala-cli run --server=false -S 3.6.3 --jvm 23 --classpath emptyClassPath.jar jsrc/legacyBug.sc

A copy or symlink of the script with .scala extension works:

scala-cli run --server=false -S 3.6.3 --jvm 23 --classpath emptyClassPath.jar jsrc/legacyBug.scala
hello legacy!

Unfortunately, that's not a viable option, because the script is then treated by Intellij as part of the project rather than a support script.

It also runs as expected if an explicit call to main() is inserted:

#!/usr/bin/env -S scala-cli run --server=false -S 3.6.3 --jvm 23 --classpath emptyClassPath.jar
LegacyBug.main(args)
object LegacyBug {
  def main(args: Array[String]): Unit = {
    printf("hello legacy!\n")
  }
}

However, the added call to main() isn't viable as a migration format for other reasons.

The scalaQ script is the only migration path I have come across so far with the following necessary characteristics:

• works with any scala3 version
• can be debugged in Intellij through a temporary symlink with .scala extension, with no editing of script content

[som-snytt]

@philwalk I'm aware that you do interesting scripting on Windows, which is not my niche; I'm on a Windows box but never leave WSL. FSR cs setup doesn't give me the scala_legacy script, though I see it in the cache. I'm hoping to avoid installing under Windows for testing, but maybe that is inevitable. My Windows 10 experience will be unsupported come October, I don't want to rock the boat.

The shebang also WFM. I see MainGenericRunner is handing off to scripting runner. So I wonder if the null only happens on Windows.

Edit: there are different options for the distro scripts, so I didn't see "-Dscala.expandjavacp=true".

Edit: the dotty build has changed at some point, so I don't even know how to test anything except scalaQ.

[philwalk]

It may well be a Windows-only problem. I didn't see it in WSL but didn't spend much time on it.

Which jdk version should I be trying to test against? I'm seeing a few test failures with jdk21. I didn't see them yesterday with either jdk11 or jdk17 (one or the other, can't remember).

there are different options for the distro scripts, so I didn't see "-Dscala.expandjavacp=true".

The scala.expandjavacp disables this warning message (a non-starter for running in production mode):

[warning] MainGenericRunner class is deprecated since Scala 3.5.0, and Scala CLI features will not work.
[warning] Please be sure to update to the Scala CLI launcher to use the new features.
[warning] Check the Scala 3.5.0 release notes to troubleshoot your installation.

[som-snytt]

jdk 17 is in CI, and for a recent PR, I switched to 17.

I forgot to write: I agree that things should just work robustly. (Without caveats for systems or how it was run.)

[Gedochao]

Just FYI, while this is a valid bug resulting in a crash, scala_legacy is a deprecated feature (being the old runner), which has been kept for ease of migration.
Efforts towards maintaining it are meant to be minimal, and it will likely be removed at some point.

[philwalk]

@Gedochao - I agree we shouldn't put much effort into legacy tools.
So far, I haven't found any viable migration path, although scala_legacy seems to work with this fix.

My preferred migration path, if it were possible, would be to be able to treat a file with .sc extension as if it had a .scala extension (perhaps via a command line option). I'm unable to just rename my scripts due to the way Intellij handles files with the .scala extension. Another possibilty would be to support the .scala behaviour but with an otherwise unused extension, maybe .scal, .sca or similar.

[Gedochao]

My preferred migration path, if it were possible, would be to be able to treat a file with .sc extension as if it had a .scala extension (perhaps via a command line option). I'm unable to just rename my scripts due to the way Intellij handles files with the .scala extension. Another possibilty would be to support the .scala behaviour but with an otherwise unused extension, maybe .scal, .sca or similar.

@philwalk I'm guessing that this part will stop being an issue after VirtusLab/scala-cli#3473 gets resolved (and we already have a fix in VirtusLab/scala-cli#3479, that's almost ready to merge at the time I'm writing this comment).

Please raise issues in the Scala CLI tracker if you run into any further problems while migrating.

[philwalk]

I'm guessing that this part will stop being an issue after VirtusLab/scala-cli#3473 gets resolved

Yes, this turned out better than I had hoped!

[hamzaremmal]

I've rearranged the labels as this issue can be reproduced without the runners and it was just one of many possible reproductions