Root package priority is wrong

[som-snytt]

Reproduction steps

Scala version: 2.13.8

Welcome to Scala 2.13.8 (OpenJDK 64-Bit Server VM, Java 17.0.2).
Type in expressions for evaluation. Or try :help.

scala> object sun { val x = 42 }
object sun

scala> import sun.x
import sun.x

scala> import jdk.CollectionConverters._
                  ^
       error: object CollectionConverters is not a member of package jdk

At scala/scala3#14781

Problem

Scala 3 agrees about jdk, but the REPLs disagree about sun.

[som-snytt]

Today's Scala 3 REPL works, but scalac only doesn't like unqualified jdk. So Scala 2 agrees with that.

It's weird that Scala 3 REPL disagrees with scalac on jdk.

[som-snytt]

Maybe sdkman is messing with me. It touches the environment. Time to finally switch to cs or CSI or whatever.

JAVA_HOME=/home/amarki/.sdkman/candidates/java/current
SBT_HOME=/home/amarki/.sdkman/candidates/sbt/current
SCALA_HOME=/home/amarki/.sdkman/candidates/scala/current

Just keeping fingers busy while listening to Don Carlos.

[som-snytt]

TIL the reason for this behavior is that jdk is presented as a definition in the outermost context. That gives it the wrong name-binding precedence.

The outermost context should perhaps be an import context (importing from the root class), like the other so-called root imports.

I tried that out, an easy change but with other adjustments required.

Possible future innovations include other special packages in root, such as for REPL.

[SethTisue]

(to fill in a bit of unstated context here that wasn't obvious to me initially,) note that there is _root_.jdk package:

scala> import _root_.jdk.
dynalink     internal     jpackage     net          security     vm
editpad      javadoc      jshell       nio          swing        xml
incubator    jfr          management   random       tools

hence the conflict.

[som-snytt]

Scala 3 has the same problem, although the ticket is currently in suspension.

I have a PR forthcoming. The fix is that root contexts should all be import contexts; currently that _root_.jdk has the priority of a definition, which is plain wrong. The change was not trivial for some reason that escapes me at the moment.

[SethTisue]

https://scala-lang.org/files/archive/spec/2.13/02-identifiers-names-and-scopes.html says:

Bindings of different kinds have precedence defined on them:

1. Definitions and declarations that are local, inherited, or made available by a package clause and also defined in the same compilation unit as the reference to them, have the highest precedence.
2. Explicit imports have the next highest precedence.
3. Wildcard imports have the next highest precedence.
4. Definitions made available by a package clause, but not also defined in the same compilation unit as the reference to them, as well as imports which are supplied by the compiler but not explicitly written in source code, have the lowest precedence.

The import of scala._, which would pick up scala.jdk, is one of the "imports which are supplied by the compiler but not explicitly written in source code" (bullet 4)

And the presence of jdk under _root_, which bullet does that fall under? It seems to me it doesn't fall under any of them; is there some other place in the spec that says why things under _root_ are in scope? I looked at Chapter 9 as well but the wordings in https://scala-lang.org/files/archive/spec/2.13/09-top-level-definitions.html#packagings don't seem — to me, anyway — to clearly specify how _root_ should behave.

currently that root.jdk has the priority of a definition, which is plain wrong

Can you spell out why that is plainly wrong? And what priority exactly are you suggesting it should have? I don't disagree necessarily, it just isn't so "plain" to me without further explanation/substantiation.

In any case, as Dale points out, it's just plain lousy that the REPL and the batch compiler don't agree here.

It would be awfully late in the Scala 2 cycle to change the behavior of the batch compiler, so I guess a PR that aligned with the REPL with the batch compiler would be welcome? (depending on exactly how the change would be justified, perhaps... I'd like to get clear on that before anybody starts hacking on this)

[som-snytt]

Sorry to have used that "obviously" language. I think the name binding is priority 4. But note that a package is not a definition; a package exists only by virtue of a definition which has the package name as a prefix of its full dotted name. (I don't know what package p { } does without trying it.)

So, to split hairs, I prefer "imports supplied by the compiler" and to implement it that way.

The alternative is to view package p has having an empty prefix which makes _root_.* visible. (That is actually close to the implementation.) But I think that is conceptually mistaken. The "empty package" as understood on the JVM is just another package whose name is empty; is it not a top-most package which might be called _root_. The visibility of the empty package has been problematic in Scala 3 for this reason.

Since _root_.jdk is not defined in the current compilation unit (in the example), whichever way you model it, its priority is 4 for purposes of name binding.

The bug is just that name lookup returns jdk as a definition in the current compilation unit.

[SethTisue]

Sorry to have used that "obviously" language

(brings to mind the old mathematician joke: https://hsm.stackexchange.com/questions/7247/in-a-popular-anecdote-who-took-20-minutes-to-decide-that-a-thing-was-obvious)

[SethTisue]

whichever way you model it, its priority is 4 for purposes of name binding.

✔️

The bug is just that name lookup returns jdk as a definition in the current compilation unit

👍