Reconsider package names

[lrytz]

Now that we change the artifact name, there's an opportunity to do binary breaking changes. (Though maybe it's a bad idea, and we should only consider it when we release a 3.0.0.)

In particular, I'm thinking of the inconsistency in the packages used for backported library classes, for example

• scala.collection.compat.immutable.LazyList, vs
• scala.util.Using

I assume the motivation to put LazyList in a compat sub-package was to avoid split packages, i.e., to ensure that this module doesn't add any classes into packages that exist in the 2.11 / 2.12 standard library? Since it's not done consistently I think we should just use the original package name for backports (like scala.util.Using does).

A small but maybe still realistic issue is that the recommended import scala.collection.compat._ brings an immutable into scope which is not scala.collection.immutable.

The scala.collection.compat package object still makes sense for extension methods (import scala.collection.compat._; xs.to(List)). Though, given that this is now library-compat, should it just be scala.compat._ instead? There might be backports of non-collection extension methods in the future. I'm not sure.

[NthPortal]

just for reference, I put LazyList in that package because ArraySeq was already there. it's possible there were reasons for ArraySeq that don't apply to LazyList, so it shouldn't have been there

[julienrf]

AFAICT, ArraySeq has been moved to package collection.compat.immutable in this PR: #31. The goal was to fix an issue with OSGI (#30). IIRC we have disabled OSGI support? So, we don’t anymore have a good reason to use this package?

[lrytz]

Thanks for digging these out, I didn't remember. My brain seems to be suppressing all memories of OSGi related work.

[SethTisue]

Now that we change the artifact name, there's an opportunity to do binary breaking changes. (Though maybe it's a bad idea, and we should only consider it when we release a 3.0.0.)

Lukas and I discussed this last week.

iirc, we sadly concluded that using a different artifact name doesn't make binary breaking changes harmless, and bumping the version number to 3.0.0 doesn't make them harmless, either. In order not to send any users to dependency hell, we must either:

1. Keep bincompat.

• Note that this was the original plan, that this library would never, ever break bincompat. We only did it going from 1.x to 2.x because we caught the motivating problem so early that we thought it was safe that nobody was using 1.x yet. But people are definitely using 2.x now.

2. make the old library and the new library have no potentially incompatible classes at all in common, so that 2.x and 3.x can be on the same classpath no problem

Lukas, is that an accurate summary of our discussion?

Option 2 seems hairy and not-worth-it to me. I'm certainly not eager to tackle it, it will annoy and confuse users, and it's not clear to me that the benefit is so great, anyway. 98% of Scala OSS has already added 2.13 to their crossbuilds, so there is value in just leaving things alone.

So I suggest we go with option 1. That would mean:

• all collections stuff continues to live in scala/collection/compat; bincompat forces this for existing stuff, and if we add more stuff, it would be confusing for everyone to have it in a different location, better to stay consistent
• everything else can live in its natural location, as we've already done with @nowarn and Using

wdyt?

[ijuma]

Changing the artifact name will potentially cause issues even with the proposed solution here. Due to transitive dependencies, one may end up with both in the classpath resulting in NoSuchMethodErrors if the newer version was used in compilation while the older version was used at runtime. Do we want to revert the artifact name change perhaps?

[lrytz]

Yeah... Seth said

make the old library and the new library have no potentially incompatible classes at all in common

That prevents classes from evolving. Let's say there's

class A { def f: Int }

in scala-collection-compat, and that it evolves to

class A { def f: Int; def g: Int }

in scala-library-compat. If it was the same module, sbt would pick the newest version. But with a different artifact name, both end up on the classpath, and behavior depends on which classfile is loaded first by scalac (compile-time) and the JVM (run-time).

Also moving a class (LazyList) from scala.collection.compat.immutable to scala.collection.immutable would not work with the two versions on the classpath. Let's say a project uses two libraries that depend on collection-compat

class Lib1(ll: comp.imm.LazyList)   // v1
class Lib2(ll: comp.imm.LazyList) { // v1
  new Lib1(ll)
}

now Lib1 migrates to library-compat

class Lib1(ll: imm.LazyList) // v2

This is not binary compatible comp.imm.LazyList is simply not the same class as imm.LazyList, so Lib2 v1 doesn't work with Lib1 v2.

---

So I only see two ways forward

• break binary compatibility
• rename back to scala-collection-compat and remain binary compatible forever

Before we decide to go back, I'd like to convince myself that breaking binary compatibility once is really not an option.

The thing is, the compat library has great potential because the Scala 2.13 standard library is locked down by the forwards binary compatibility constraint. Scala 2.13 will have a very very long lifetime and 3.0 is planned to use the 2.13 library too. The compat library allows extending the standard library during that time. It's quite unfortunate that we have to explain to all users "for historical reasons, this library has a weird name and the package structure is inconsistent".

Do we have a chance to find the direct users of scala-collection-compat and somehow make a coordinated effort to move everyone to the new artifact that breaks bin compat?

[julienrf]

Maybe one solution to rename the artifact without ending up with two artifacts in the classpath, would be to keep scala-collection-compat, and to create an empty artifact named scala-library-compat that depends on scala-collection-compat. This way, dependency resolvers would eventually only select one scala-collection-compat artifact, even if your project depends on two libraries that depend on different versions of scala-collection-compat/scala-library-compat.

That doesn’t solve the question of breaking binary compatibility or not, though…

[ijuma]

Maybe the library for extending 2.13/3.0 could be scala-library-compat. It could use different package/class names and be a longer term thing. Or maybe the name should not even include compat, if that's the goal.

scala-collection-compat would continue to be used for bridging 2.11, 2.12 and 2.13. For reference, Apache Kafka no longer supports Scala 2.11. If/when we add support for 3.0, we will likely drop support for 2.12. This is just one project, but the overhead of supporting more than two versions of Scala is pretty high and I don't think it's uncommon for projects to stick to 2 versions.

[lrytz]

In the end, I don't think breaking binary compatibility is an option. Browsing through GitHub search, the library is used too widely, and fixing dependency conflicts will be too hard for users (figure out which of their depdendencies is built against an old collection-compat, check if there's a new version, ping maintainers, etc). So the packages have to remain what they are; as Seth said

• collections stuff continues to live in scala/collection/compat, also new backports
• everything else can live in its natural location, as we've already done with @nowarn and Using

For changing the artifact name, Julien's idea sounds plausible to me (but we'd have to do some testing). So we could release a new scala-library-compat:1.0.0 that is binary compatible with scala-collection-compat:2.1.6. To avoid having both scala-library-compat:1.0.0 and scala-collection-compat:2.1.6 on the classpath, the new scala-library-compat:1.0.0 would depend on a completely empty scala-collection-compat:2.2.0.

About including new classes for 2.13, we can either do it in this library, or in a completely new one. I'm fine either way. The 2.13 artifacts of scala-collection-compat are currently empty except for two package objects (scala.collection.compat and scala.collection.compat.immutable) that contain type and val aliases, which is a clean enough starting point. If we do that, I think we should rename the library to scala-library-future.

To summarize, let's vote on these options (please comment for feedback / other suggestions):

• 😄 rename back to scala-collection-compat. Continue adding backports for 2.11 and 2.12, keep 2.13 empty.
  Pro: least breaking change, nobody needs to change the dependency.
  Con: misleading name, the library contains backports outside collections.
• 🎉 keep scala-library-compat or rename to scala-library-backports. Continue adding backports for 2.11 and 2.12, keep 2.13 empty.
  Pro: better library name.
  Con: users need to find the new artifact name (breaks scala steward), ghost dependency on empty scala-collection-compat:2.1.7.
• 🚀 move to scala-library-future. Continue adding backports, also add new features for 2.13/2.12/2.11.
  Pro: a single place for standard library extensions.
  Con: same as above.

If we pick 😄 or 🎉, we can at any point start a new library scala-library-future that contains extensions for 2.13 (and backports of them for 2.12 / 2.11).

[dwijnand]

• users need to find the new artifact name (breaks scala steward)

Feels viable to teach Scala Steward this, at worst non-generically.

[NthPortal]

IMO, for maximum ease, we should keep the new scala-library-compat at the same version as scala-collection-compat (i.e. start it at 2.2.0).

To avoid having both scala-library-compat:1.0.0 and scala-collection-compat:2.1.6 on the classpath, the new scala-library-compat:1.0.0 would depend on a completely empty scala-collection-compat:2.2.0.

something about that doesn't sound right to me. the "outer" dependency (i.e. the one that depends on the other) needs to be the one that's empty. so either collection-compat is empty, and depends on library-compat, or library-compat is empty, and depends on collection-compat. as described, if someone upgrades to collection-compat:2.2.0, they will end up with a broken dependency lacking contents (and it would thus, technically speaking, not be binary-compatible).

[julienrf]

IMO, for maximum ease, we should keep the new scala-library-compat at the same version as scala-collection-compat (i.e. start it at 2.2.0).

Why would that make things easier? Dependency resolution systems (e.g., coursier) will consider anyway that the two artifacts, scala-library-compat et scala-collection-compat are independent from each other so they won’t even compare their revision numbers.

as described, if someone upgrades to collection-compat:2.2.0, they will end up with a broken dependency lacking contents (and it would thus, technically speaking, not be binary-compatible).

Right, good catch! My initial proposal was to keep the content of scala-collection-compat as it is (so that it remains backward binary compatible in the next versions, and to put new stuff (unrelated to the collections) into a new artifact, scala-library-compat, that depends on scala-collection-compat (or, we could put everything into collection-compat and have library-compat an empty artifact that depends on it). Does that sound correct?

[NthPortal]

Why would that make things easier?

my thought was that it makes it easier for the end user to switch from the "deprecated" collection-compat to the preferred library-compat whenever they want to, since they will have the same version