flesh out JDK 11 and 12 community builds

[SethTisue]

in #742 we got the build up and running, all projects extracting, and enough green projects (18 green) to be sure that basics are working

next steps:

• fix any new 11-only failures that are making the 11 build worse than the old 9 & 10 builds, the big glaring one is the scalacheck failure (because .lines)
• then resume making gradual progress towards an all-green build, as we've been doing for much of 2018

[SethTisue]

I fixed these blockers: scalacheck, scalatest, utest, scala-java8-compat.

a new run is now in progress at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/18/consoleFull and will hopefully have substantially more than 18 green projects

@smarter joked that "every Scala codebase ever" is going to hit scala/bug#11125, and it seems that's not far from the truth.

modifying projects one at a time to replace foo.lines with Predef.augmentString(foo).lines feels like the kind of work Judge Wills sentenced Oscar Wilde to. perhaps the problem can be solved community-build-wide with appendScalacOptions -release 8. I'll try it.

[SethTisue]

with -release 8 multiple projects fail to compile because during Scaladoc generation we get the same "java.lang.AssertionError: assertion failed: type AnyRef in java.lang" errors @retronym and the Akka folks already saw back in March at akka/akka#24711 (comment)

[SethTisue]

"As a workaround, try removing -release 8 from the scalacOptions in doc", suggests @retronym. our current addScalacOptions mechanism isn't customizable that way, but probably I could do it project-by-project (which isn't so painful when it's only our own config file I'm modifying, not having to push commits to lots of forked repos)

[SethTisue]

60 projects are green now. top ​blockers: ​87 ​specs2, ​81 ​macro-paradise, ​22 ​fastparse, ​21 ​akka

I'll need to go through the failures one by one and see if any them represent regressions in 2.12.7.

[retronym]

Can we disable scaladoc generation globally and conveniently?

[SethTisue]

@retronym good thought, I'll come back to it later.

the 14 failures at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/18/consoleFull:

• specs2, expecty, fastparse, macro-paradise, pprint, scala-partest, scala-refactoring, scalamock: consider rename StringOps#lines. conflict jdk11 java.lang.String#lines bug#11125
• scodec-bits: newly ambiguous methods on ByteBuffer because Oracle added new overloads
• akka: [akka] akka.actor.dungeon.SerializationCheckFailedException: Failed to serialize and deserialize message of type java.util.BitSet for testing. To avoid this error, either disable 'akka.actor.serialize-messages', mark the message with 'akka.actor.NoSerializationVerificationNeeded', or configure serialization to support this message
• pcplod: scala.reflect.internal.MissingRequirementError: object scala.annotation.Annotation in compiler mirror not found

the only one of that looks maybe a bit worrisome is the pcplod one, but pcplod is an unusual project, I don't think we need to worry about it for 2.12.7

TL;DR looks good for 2.12.7

[adriaanm]

Thanks Seth! I know it’s late over there.

[SethTisue]

etorreborre/specs2#710 should fix specs2

[SethTisue]

scalacommunitybuild/paradise@e4b6a5f should fix macro paradise

[xuwei-k]

scodec-bits: newly ambiguous methods on ByteBuffer because Oracle added new overloads

scodec/scodec-bits#98

[xuwei-k]

alexarchambault/case-app#92

[xuwei-k]

playframework/play-doc#40

[xuwei-k]

typelevel/fs2#1278

[SethTisue]

thank you, Yoshida-san, for continuing to chip away at this. I'll keep at it as well :-)

https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/57/ shows the current top blockers as:

BLOCKERS:
22 fastparse
21 akka
20 ssl-config
14 pprint
14 scalatags
13 expecty
4 fs2
4 case-app
3 slick
3 twitter-util
3 scalamock
1 blaze

[xuwei-k]

• add jdk11 test eed3si9n/expecty#6
• add jdk11 test atnos-org/eff#152

[xuwei-k]

com-lihaoyi/fastparse#196

[SethTisue]

lightbend/ssl-config#128 (just an issue, not a PR)

[SethTisue]

com-lihaoyi/scalatags#184

[SethTisue]

coursier/coursier#935 (just an issue, not a PR)

[SethTisue]

scalameta: f4dfa83

[SethTisue]

scopt/scopt#207

[exoego]

lihaoyi/Scalatex#67

[SethTisue]

139 green, up from 120 two weeks ago!

in some cases JDK 11 support PRs have been merged, in other cases I've (impatiently) changed our config to point to the PR branch

status as of https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/105/ is:

SUCCESS 139 FAILED 13 DID NOT RUN 37 TOTAL 189
UNEXPECTED 0
BLOCKERS:
20 ssl-config
13 metaconfig
4 fs2
3 coursier
3 slick
3 twitter-util
3 scalamock
1 blaze

from report/Report.scala:

  val expectedToFail: Set[String] =
    System.getProperty("java.specification.version") match {
      case "1.8" =>
        Set()
      case _ =>
        Set(
          "blaze",  // test failure (org.http4s.blaze.pipeline.stages.SSLStageSpec)
          "breeze",  // "object istack is not a member of package com.sun"
          "case-app",  // needs scala/bug#11125 workaround
          "coursier",  // needs scala/bug#11125 workaround
          "fs2",  // new overloads on JDK 11 need disambiguation
          "kxbmap-configs",  // "sjsonnew.DeserializationException: Field not found: $type"
          "metaconfig",  // needs scala/bug#11125 workaround
          "scala-refactoring",  // needs scala/bug#11125 workaround?
          "scalamock",  // needs scala/bug#11125 workaround
          "scalatex",  // scripted-plugin not found?
          "slick",  // "object bind is not a member of package javax.xml"
          "ssl-config", // https://github.com/lightbend/ssl-config/issues/98
          "twitter-util",  // "javadoc: error - invalid flag: -d"
        )
    }

[xuwei-k]

"case-app", // needs scala/bug#11125 workaround

fixed in master alexarchambault/case-app@5023eaa

https://github.com/scala/community-builds/blob/01a80322b6a74d79a4181d56eb6c6200e813f050/configs/project-refs.conf#L27

"fs2", // new overloads on JDK 11 need disambiguation

typelevel/fs2@226ff68#diff-736c8aaa74103281c32f371842cf994bR41

https://github.com/scala/community-builds/blob/01a80322b6a74d79a4181d56eb6c6200e813f050/configs/project-refs.conf#L53

"scalamock", // needs scala/bug#11125 workaround

scalamock/scalamock@ff52907

https://github.com/scala/community-builds/blob/01a80322b6a74d79a4181d56eb6c6200e813f050/configs/project-refs.conf#L145

"twitter-util", // "javadoc: error - invalid flag: -d"

sbt/sbt#4350

[SethTisue]

@xuwei-k thanks for all this info. I've handled them as follows:

• case-app: cherry-picked the change onto our fork
• fs2: rather than mess with forking, let's move to 1.0.0 and get the fix that way. new ticket: move fs2 to 1.0.0 #804
• scalamock: forked it and cherry-picked the change (f8d3fe7)
• twitter-util: for now, just added a comment about it (f8d3fe7). let Eugene know on the sbt ticket that a fix would be nice to have

Dale hopes to fix ssl-config soon, lightbend/ssl-config#98

[SethTisue]

https://github.com/olafurpg/metaconfig/pull/61 , b08a23d

[exoego]

If my investigation and understanding is correct, JDK 9+ support on slick requires

• Oracle XE 18c that will be released on Octoberer 2018 or later.
• Public Docker image for Oracle XE 18c

[SethTisue]

scalameta/scalafmt#1304

[SethTisue]

scalafmt is green at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/126/, we're up to 148 green now

BLOCKERS:
20 ssl-config
4 fs2
3 coursier
3 slick
3 twitter-util
3 sbt-util
1 blaze

I'm working on ssl-config at #808 and fs2 at #805

sbt-util is an easy fix if someone wants to grab that one

[SethTisue]

ssl-config, fs2, blaze went green

I'll ask the Akka folks about the akka-streams compile error

SUCCESS 153 FAILED 11 DID NOT RUN 24 TOTAL 188
BLOCKERS:
15 akka-stream
3 coursier
3 slick
3 twitter-util
3 sbt-util

[SethTisue]

I haven't had time to work on this in the past month. The blockers list remains as above.

[SethTisue]

https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk11-integrate-community-build/231/

akka-stream is green now (I finally figured out a -release 9 needed to be -release 11 on some machines including our Jenkins, even though it worked on some other setups)

that brings us up to 165 green projects total, which is very satisfying. no major blockers remain

SUCCESS 165 FAILED 12 DID NOT RUN 14 TOTAL 191
UNEXPECTED 5
BLOCKERS:
3 coursier
3 slick
3 twitter-util
3 playframework
3 sbt-util
1 akka-http-cors

I tried unfreezing coursier (#823) but ran into trouble with downstream projects, I'll have to come back to that

the next failure to look at is playframework, @dwijnand want to take a look at the errors?

[dwijnand]

@SethTisue SethTisue/playframework#1

[dwijnand]

Actually scalacommunitybuild/playframework#1.

[SethTisue]

lightbend/ssl-config#142 / playframework/play-meta#4 makes the Play situation messy, so I'm just going to sit tight and wait for that to get sorted out

[SethTisue]

if anyone else is watching this ticket and would like to help out, the current failures list is: scala-refactoring, compiler-bridge-1-0, twitter-util, slick, kxbmap-configs, coursier, sconfig, scala-debugger, breeze, doobie, playframework, sbt-util

and what's blocking downstream projects is:

3 coursier
3 slick
3 twitter-util
3 playframework
3 sbt-util

[SethTisue]

it's time to start testing on JDK 12, too: #862

[SethTisue]

#862 is complete, all projects extract. I've broadened the scope of this ticket to cover both 11 and 12

[SethTisue]

the JDK 12 build is now fully on par with the JDK 11 one; both have 165 green projects

[SethTisue]

the JDK 11 and 12 builds had fallen behind the 8 one somewhat while my attention was elsewhere

but with the Scala 2.12.9 release coming up, it was important to return to it, so after some fixes, the 11 and 12 builds are green again (not counting a handful of known failures that I've evaluated and/or reported to upstream maintainers)

[SethTisue]

closing, as keeping the 11 & 12 builds in shape is now part of normal maintenance.

[SethTisue]

"As a workaround, try removing -release 8 from the scalacOptions in doc", suggests @retronym. our current addScalacOptions mechanism isn't customizable that way

for e.g. spray-json I was able to do

    extra.commands: ${vars.default-commands} [
      """set sprayJsonJVM / Compile / doc / scalacOptions --= Seq("-release", "8")"""
    ]

but it might be harder in a multi-project build

I finally got around to opening a ticket-of-record on the underlying bug: scala/bug#11682