2.12.x consistently failing bootstrap (OSGi tests)

[SethTisue]

last green run was https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-bootstrap/640/consoleFull, scala rev 4a487ca5d0da7925fcd23772e25cc10d8167a351

first red run was https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-bootstrap/641/consoleFull, scala rev 066637781d42d8c512abaf7496e679e0e9f250e6 see comment below

[error] 2 of 11 test tasks failed:
[error] - osgiTestFelix/test
[error] - osgiTestEclipse/test

@lrytz the cause appears to be scala/scala#5482

[lrytz]

uh oh :( will take a look!

[lrytz]

i haven't managed to reproduce it yet locally

[lrytz]

comparing the last good and first bad. good:

[warn] bnd: Invalid package name: 'repl-jline.properties'
[warn] bnd: Invalid package name: 'scala-asm-5.1.0-scala-1.jar'
[warn] bnd: Invalid package name: 'scala-buildcharacter.properties'
[info] Compiling 5 Scala sources to /home/jenkins/workspace/scala-2.12.x-integrate-bootstrap/target/osgiTestFelix/test-classes...
[info] Test run started
[info] Test tools.test.osgi.libonly.BasicLibraryTest.everythingLoads started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 0.766s
[info] Test run started
[info] Test tools.test.osgi.reflection.basic.BasicReflectionTest.basicMirrorThroughOsgi started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 1.714s
[info] Test run started
[info] Test tools.test.osgi.BasicTest.everythingLoads started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 0.32s
[info] Test run started
[info] Test tools.test.osgi.reflection.toolbox.ReflectionToolBoxTest.basicMirrorThroughOsgi started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 4.552s
[info] Passed: Total 4, Failed 0, Errors 0, Passed 4
[info] Compiling 5 Scala sources to /home/jenkins/workspace/scala-2.12.x-integrate-bootstrap/target/osgiTestEclipse/test-classes...
[info] Test run started
[info] Test tools.test.osgi.libonly.BasicLibraryTest.everythingLoads started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 0.815s
[info] Test run started
[info] Test tools.test.osgi.reflection.basic.BasicReflectionTest.basicMirrorThroughOsgi started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 1.693s
[info] Test run started
[info] Test tools.test.osgi.BasicTest.everythingLoads started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 0.369s
[info] Test run started
[info] Test tools.test.osgi.reflection.toolbox.ReflectionToolBoxTest.basicMirrorThroughOsgi started
[info] Test run finished: 0 failed, 0 ignored, 1 total, 4.742s
[info] Passed: Total 4, Failed 0, Errors 0, Passed 4
[info] Packaging /home/jenkins/workspace/scala-2.12.x-integrate-bootstrap/build/pack/lib/scala-library.jar ...
[info] Done packaging.
[info] Resolving org.scala-lang#scala-library;2.12.0-RC1 ...

the corresponding place in the bad one:

[warn] bnd: Invalid package name: 'repl-jline.properties'
[warn] bnd: Invalid package name: 'scala-asm-5.1.0-scala-1.jar'
[warn] bnd: Invalid package name: 'scala-buildcharacter.properties'
[warn]  module not found: org.scala-lang.modules#scala-swing_2.12.0-0666377-nightly;2.0.0-M2
[warn] ==== private-repo: tried
[warn]   https://scala-ci.typesafe.com/artifactory/scala-release-temp/org/scala-lang/modules/scala-swing_2.12.0-0666377-nightly/2.0.0-M2/scala-swing_2.12.0-0666377-nightly-2.0.0-M2.pom
[warn] ==== jcenter-cache: tried
[warn]   https://scala-ci.typesafe.com/artifactory/jcenter/org/scala-lang/modules/scala-swing_2.12.0-0666377-nightly/2.0.0-M2/scala-swing_2.12.0-0666377-nightly-2.0.0-M2.pom
[warn] ==== typesafe-ivy-releases: tried
[warn]   https://repo.lightbend.com/typesafe/ivy-releases/org.scala-lang.modules/scala-swing_2.12.0-0666377-nightly/2.0.0-M2/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn]   https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/org.scala-lang.modules/scala-swing_2.12.0-0666377-nightly/2.0.0-M2/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/scala-lang/modules/scala-swing_2.12.0-0666377-nightly/2.0.0-M2/scala-swing_2.12.0-0666377-nightly-2.0.0-M2.pom
[warn] ==== local: tried
[warn]   /home/jenkins/workspace/scala-2.12.x-integrate-bootstrap/ivy2/local/org.scala-lang.modules/scala-swing_2.12.0-0666377-nightly/2.0.0-M2/ivys/ivy.xml
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.scala-lang.modules#scala-swing_2.12.0-0666377-nightly;2.0.0-M2: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn] 
[warn]  Note: Unresolved dependencies path:
[warn]      org.scala-lang.modules:scala-swing_2.12.0-0666377-nightly:2.0.0-M2 (/home/jenkins/workspace/scala-2.12.x-integrate-bootstrap/build.sbt#L868)
[warn]        +- org.scala-lang:dist:2.12.0-0666377-nightly
[info] Packaging /home/jenkins/workspace/scala-2.12.x-integrate-bootstrap/build/pack/lib/scala-library.jar ...
[info] Done packaging.
[info] Resolving org.scala-lang#scala-library;2.12.0-RC1 ...

[adriaanm]

Not sure, but could that unresolved dependencies message be related to the check whether we already published the module or not?

[lrytz]

I don't know, but in the failing run, Compiling 5 Scala sources to .../osgiTestFelix and Compiling 5 Scala sources to ../osgiTestEclipse are missing.

[lrytz]

another thing i don't understand: if i look at the repo, there's "scala-swing_2.12", not a cross versioned thing. https://scala-ci.typesafe.com/artifactory/scala-release-temp/org/scala-lang/modules/

the build seems to be looking for scala-swing_2.12.0-0666377-nightly

not sure if that's related.

[SethTisue]

just a guess, but might scala/scala#5389 help?

[SethTisue]

@szeiger @lrytz might scala/scala#5379 have something to do with this? never mind

[SethTisue]

another thing i don't understand: if i look at the repo, there's "scala-swing_2.12", not a cross versioned thing. https://scala-ci.typesafe.com/artifactory/scala-release-temp/org/scala-lang/modules/

the build seems to be looking for scala-swing_2.12.0-0666377-nightly

I'm seeing something possibly related in the community build where lift-json started failing every time because it can't find a scalap jar with a nightly version number. at https://scala-ci.typesafe.com/artifactory/scala-release-temp/org/scala-lang/scalap/ there's only "2.12.0"

this doesn't seem like it can possibly have to do with 5482, so I'm looking around for other causes. the mysterious thing to me is why stuff only just started failing now. it's not like those directories have any older 2.12 nightly artifacts in them

[SethTisue]

good news: https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-bootstrap/643/ passed and published artifacts to https://scala-ci.typesafe.com/artifactory/scala-release-temp/org/scala-lang/

[SethTisue]

I think the the story here is simply:

• releasing 2.12.0 somehow removed all the old 2.12.0 nightlies from our Artifactory
• the missing artifacts caused various effects downstream
• everything's fine again now that we're into the 2.12.1 series

[SethTisue]

I just noticed that I got the run numbers wrong above. it should have been:

last green run was https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-bootstrap/638/consoleFull, scala rev 4a487ca5d0da7925fcd23772e25cc10d8167a351

first red run was https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-bootstrap/639/consoleFull, scala rev 066637781d42d8c512abaf7496e679e0e9f250e6

I had the Scala revs right, but our working hypothesis is that those aren't important, the timing of the 2.12.0 release is. And in fact, 638 was before Adriaan built 2.12.0 and 639 was after.

[SethTisue]

On Slack, Lukas points out "the bootstrap script does have code to delete from the repo first: removeExistingBuilds()"

and yeah, select(.uri | contains(\"$SCALA_VER\")) sure looks like suspicious code that would delete every 2.12.0 nightly along with 2.12.0 itself