Clarify spec on precedence of definitions in the same package and imports supplied by the compiler

[unkarjedy]

https://scala-lang.org/files/archive/spec/2.13/02-identifiers-names-and-scopes.html

Currently, spec says:

Bindings of different kinds have precedence defined on them:
...
4. Definitions made available by a package clause, but not also defined in the same compilation unit as the reference to them, as well as imports which are supplied by the compiler but not explicitly written in source code, have the lowest precedence.\

It sounds like definitions in the same package and imports supplied by the compiler have the same precedence.

But looks like definitions in the same package have a higher precedence.
Otherwise, I would expect this code to produce some ambiguity error.
But it compiles and shows that definition in the same package is preferred

I think we could split spec item 4. into two items 4. and 5.
If you agree I could make a PR

[som-snytt]

I'm not sure what the picture is saying, but the spec goes on to say:

A binding in some inner scope shadows bindings of lower precedence in the same scope as well as bindings of the same or lower precedence in outer scopes.

The so-called "root imports" or "root contexts" are the outermost scopes, so they are shadowed by anything in source code. They work like an enclosing package in which your code is nested, as though you had written

package _root_
package mystuff

so that anything in the root package is visible in mystuff (but may be shadowed by definitions in mystuff).

I think the spec already says this, but of course clarifications are welcome. The text and its implementation have seen several iterations.

The root contexts are also nested, so the order of -Yimports matters for shadowing.

[unkarjedy]

The so-called "root imports" or "root contexts" are the outermost scopes, so they are shadowed by anything in source code. They work like an enclosing package in which your code is nested, as though you had written
so that anything in the root package is visible in mystuff (but may be shadowed by definitions in mystuff).

Maybe I misunderstood something, but:
scala.Responder comes from _root_.scala._, not from _root_._

Here its said that:

Every compilation unit implicitly imports the following packages, in the given order:

• the package java.lang,
• the package scala, and
• the object scala.Predef, unless there is an explicit top-level import that references scala.Predef.

Members of a later import in that order hide members of an earlier import.

I assume that "imports which are supplied by the compiler" == "implicit imports"
I read this (wrongly?) as "implicit imports are implicitly inserted in the beginning of the packaging":

package myStuff

import java.lang._
import scala._
import scala.Predef._
//explicit imports go here...

I can guess that combining this information

The so-called "root imports" or "root contexts" are the outermost scopes, so they are shadowed by anything in source code. They work like an enclosing package in which your code is nested, as though you had written

and

Every compilation unit implicitly imports the following packages, in the given order:

The resulting code is something like:

package _root_

import java.lang._
import scala._
import scala.Predef._

package myStuff

//explicit imports go here...

If so, then I can't see how it can be inferred from the current spec.
It looks more like an implementation detail.

[som-snytt]

No, they are not like imports, which is why "root imports" is a misnomer. It is as I described, which is more precise than the spec has it in the section about the "preamble". "These imports are taken as lowest precedence, so that they are always shadowed by user code, which may contain competing imports and definitions. They also increase the nesting depth as shown, so that later imports shadow earlier ones." Those words attempt to describe the behavior, but it's wrong. It's really just as though each root context were an enclosing package. Maybe that is the section that needs clarification.

[unkarjedy]

It's really just as though each root context were an enclosing package

This also doesn't look like the best description.
If those "implicit imports" were treated as enclosing packages that would mean that all user definitions would have an implicit package prefix java.util.scala.Predef

It's something in between...
Like imports in a preamble to "every package", not "every source file"

The compiler supplies imports in a preamble to every source file.
This preamble conceptually has the following form, where braces indicate nested scopes:

[som-snytt]

It's compiler magic, but the mechanism is just an enclosing context. You could think of it like the "empty package", a package with no package prefix. I would argue that the empty package is equally strange.

[unkarjedy]

On the whole, I don't think that it will hurt splitting this:

4. Definitions made available by a package clause, but not also defined in the same compilation unit as the reference to them, as well as imports which are supplied by the compiler but not explicitly written in source code, have the lowest precedence.\

into two items, in order it's consistent with what's written latter:

The compiler supplies imports in a preamble to every source file... These imports are taken as the lowest precedence

WDYT?

[som-snytt]

I see what you mean, that it is explanatory, but it's also true that the spec covers the behavior.

That is, root context is not lower precedence, it's an outer scope. Normal shadowing is at work.

However, it really is compiler magic, so maybe what matters is whatever metaphor makes it palatable.

It's also nice to minimize the number of rules.

The other question is maybe the spec is ok, but there is a FAQ or overview doc that could clarify how it works.

I should add that I considered what you suggest, which is plausible, but I realized it's not necessary. That was when I contributed a fix in this area.

I'd suggest that if you make a 5th precedence level for it, also edit the "import" analogy. Just call it a magical introduction of symbols. You can't write it in code, so don't try to explain it in ordinary terms of the language. I think Lukas made a similar point back then.

I think -Yimports is underused, and maybe will never appear in Scala 3. But if you use cats everywhere, why not put it in a root context? Scala 3 is all about contextual abstraction. Why should I have to type import everywhere?

[unkarjedy]

BTW I wonder what is the motivation behind the fact that wildcards imports should have higher precedence than "Definitions made available by a package clause".

It's different from Java behavior and complicates some refactoring scenarios.
Here are two almost identical examples in Java and Scala:

package org.example;

import org.example.MyData;
import some.library.*;

public class JavaExample {
  public static void main(String[] args) {
    System.out.println(MyData.class);
    System.out.println(MyDataOther.class);
  }
}

package org.example

import org.example.MyData
import some.library._

object ScalaExample {
  def main(args: Array[String]): Unit = {
    println(classOf[MyData])
    println(classOf[MyDataOther])
  }
}

See this redundant import org.example.MyData?
(It's not "unused" because it IS used during resolve by the compiler, but it's "redundant" because it's in the same package)

In Java, we can safely remove the import because it's already available in current package:

In Scala 2.13, however, we can't do this right away.
We need to analyze all wildcard imports in the current scope and all parent scopes (in case of local import)
and see if it has any name collisions with MyData.
Because if there is some.library.MyData then we can't delete import org.example.MyData because MyData will resolve to some.library.MyData then.

Note, that such code (with imports from the same package) can easily be observed during Move refactoring.

Before Scala 2.13 it worked the same way as in Java, though it was against the specification.
As I understand, at some point in time it was decided to fix the implementation.
Now I wonder maybe the specification should have been fixed instead.

[som-snytt]

I can answer that Odersky is confident in how it works (and it works the same in Scala 3), with the reasoning: it should prefer what I wrote in my source file, not what someone else wrote somewhere else.

In your example, the unwanted name binding in the wildcard import should have been excluded.

import some.library.{MyData => _, _}

Some people are averse to wildcard imports, and certainly implicit resolution in Scala 3 is designed to reduce problems due to name collisions.

[som-snytt]

Related bug #12566 where Seth asks how is binding of root package names defined?

That related bug concerns the precedence of _root_.jdk and scala.jdk; I argue that scala.jdk should shadow the root package.

(Worth adding that there is no mention of "class path", which matters for classes in the empty package; packages are open, though won't that change in the modular era? That is an implementation detail about class paths.)

A further observation on the last example about the difference between Java and Scala: Java has flat packages, so import really means import from other packages. (Or statically.) Scala's nested packages means lexical scoping of imports is meaningful.

To say, "If I refactor imports, then I'll have to look at all the imports," is like saying, "If I refactor a variable name, I have to look at all the names [that might conflict]."

[som-snytt]

The "enclosing package" analogy for "root contexts" breaks down because package-private members of a predef object are selectively available to user code depending on usual access rules. (Which makes them more like an import, or "in-between", as noted earlier.)