lots of resolver errors in build logs (maven.twttr.com)

[SethTisue]

lukas [2:47 PM]
seth, it's a detail, but do you know why this line keeps appearing in the community build log?
[...] [error] SERVER ERROR: Service Temporarily Unavailable url=https://maven.twttr.com/<some>/<maven>/<artifact>/<path>

​sethtisue [2:49 PM]
it started when I added a resolver needed by twitter-util
I’ve asked Toni if it’s possible to add a resolver just to a single project, Toni says no

lukas [2:49 PM]
is there an order in the resolvers list? then it could just come last?

sethtisue [2:50 PM]
it is already last or very close. it’s mysterious to me
this falls under the general heading of #152
it could be that we need to mirror more of the resolvers in the Artifactory config on scala-ci, but the idea of having to mess with that depresses me

[ghost]

Following up on chat on https://gitter.im/typelevel/general?at=57136f222c9711166432a27e

I'm seeing constant local build time of around 2h 30 min, mainly achieved by using artifactory with virtual repositories, as per #152 (comment)

The setup is really rather easy:

Just add the extra repositories listed in common.conf to artifactory as Maven/Ivy/sbt remote repositories (Make sure "Suppress POM Consistency Checks" is ticked). At this point, you have just proxied some extra repos.

Then create three virtual repositories, such as (CB-Ivy, CB-Maven, CB-Sbt) and add the current repositories listed in common.conf to the appropriate Virtual repo, in the same order. The idea here is that all the drop down through the repos is "pre-calculated"

Your repositories in common.conf is then something like:

 local
  cb-ivy:   YOUR-URL/artifactory/CB-Ivy, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
  cb-maven: YOUR-URL/artifactory/CB-Maven
  cb-sbt:   YOUR-URL/artifactory/CB-Sbt, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext] 

If needed, you could always create the Virtual repos per config - eg "CB-Ivy_2.12.x" and so on.

I already had Artifactory setup, but not used by default and it did help adding it to dbuild, but the dbuilds were still really slow. ( as in Ctrl-C slow...).

Adding the Virtual repos was the cure :)

[adriaanm]

Should be resolved by #243 and #244 (via scala/scala-jenkins-infra#171)

[adriaanm]

Note that this moves all the resolver config into artifactory, whose config isn't under source control... (scala/scala-jenkins-infra#172)

[adriaanm]

ugh, well, it looks like now the slowness turned into resolver issues

[adriaanm]

Looking into this, it seems the maven.twttr.com repo is returning jars using the text/html mimetype? E.g., https://maven.twttr.com/com/twitter/common/objectsize/0.0.10/objectsize-0.0.10.jar is showing up as text in my browser, and artifactory refuses to cache it. (https://scala-ci.typesafe.com/artifactory/twitter-maven/com/twitter/common/objectsize/0.0.10/objectsize-0.0.10.jar says:

{
  "errors" : [ {
    "status" : 409,
    "message" : "Error fetching https://maven.twttr.com/com/twitter/common/objectsize/0.0.10/objectsize-0.0.10.jar: Mismatching mime types found in request: [application/java-archive] and response: text/html, and this repo blocks text/html on mismatch (remote response: 409: null)"
  } ]
}

)

[SethTisue]

@stuhood is this something someone at Twitter could address? serving up artifacts with the wrong MIME type seems like something that could affect a variety of downstream tools, not just the Scala community build in particular

[SethTisue]

the problem on the Twitter end still exists afaik (I didn't check again just now), but we don't care so much at our end now since #243, so I'm closing this

[SethTisue]

fyi @mosesn @kevinoliver

[mosesn]

@SethTisue thanks, we're aware of the problem but don't have an excellent solution yet.

[SethTisue]

recently: a team at Twitter "took it upon themselves to migrate their dependencies to maven central. The need to use maven.twttr.com should go away entirely soon"

Adriaan: "We'll drop the maven.twttr.com repo from our artifactory cache once the move is complete"

new ticket: scala/scala-jenkins-infra#197