Skip to content

Upgrade to sbt 1 #3441

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 49 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
6aafb69
sbt 1.0.0-M6
smarter Jun 6, 2017
4ce83b8
Upgrade to sbt 1.0.0-M6
smarter Jun 6, 2017
c91171d
Fix sbt bridge
ttreyer May 10, 2017
024f5fb
Add method SymDenotation.isTopLevelClass
ttreyer May 10, 2017
e1018d1
Remove SimpleType
ttreyer May 15, 2017
7865186
Update bridge for compatibility with Zinc 1.0.0-X15
ttreyer May 21, 2017
3f8a49c
Fix for zinc 1.0.0-X16
smarter Jun 6, 2017
848e41b
ExtractAPI: fix children of sealed classes
smarter Jun 6, 2017
bfd2b0f
ExtractAPI: add missing startSource callback
smarter Jun 6, 2017
d3f50a4
Fix various issues
smarter Jun 6, 2017
f02dacb
Propagate class files through GenBCode pipeline
ttreyer Jun 7, 2017
d7dba97
Catch FileConflictException creating class files
ttreyer Jun 7, 2017
fb9c611
Adapt to latest Zinc
Duhemm Oct 12, 2017
aef68c2
Compile only the bridge, not the whole compiler
Duhemm Oct 28, 2017
0f13174
Adapt to new main class detection in Zinc 1
Duhemm Oct 20, 2017
323b451
Re-enable `dist-*` projects
Duhemm Oct 23, 2017
c05353a
Fix `bin/dot*` scripts
Duhemm Oct 23, 2017
4c91ed2
Fix `source-dependencies/abstract-type-override`
Duhemm Oct 20, 2017
2cf0abe
Fix `source-dependencies/transitive-memberRef`
Duhemm Oct 20, 2017
f232fbe
Disable `source-dependencies/inherited-deps-java`
Duhemm Oct 20, 2017
494beb9
Fix test `source-dependencies/compactify`
Duhemm Oct 20, 2017
b749648
Mark test `source-dependencies/binary` pending
Duhemm Oct 20, 2017
dd7841d
Fix test source-dependencies/export-jars
Duhemm Oct 20, 2017
98c9e8d
Update test `source-dependencies/import-class`
Duhemm Nov 7, 2017
479c6c2
Fix test source-dependencies/trait-member-modified
Duhemm Oct 20, 2017
240e133
Fix test source-dependencies/java-analyis-serialization-error
Duhemm Oct 20, 2017
08e09b8
Fix test source-dependencies/canon
Duhemm Oct 20, 2017
939dd31
Fix test source-dependencies/restore-classes
Duhemm Oct 20, 2017
4ee6fd5
Fix test source-dependencies/type-alias
Duhemm Oct 20, 2017
159a19e
Fix test source-dependencies/typeref-only
Duhemm Oct 20, 2017
149c8b1
Fix test source-dependencies/less-inter-inv-java
Duhemm Oct 20, 2017
389cdcc
Fix test source-dependencies/backtick-qupted-names
Duhemm Oct 20, 2017
ca77743
Fix test source-dependencies/replace-test-a
Duhemm Oct 20, 2017
e4f092d
Fix test source-dependencies/inherited_type_params
Duhemm Oct 20, 2017
ce237c7
Fix test source-dependencies/same-file-used-names
Duhemm Oct 20, 2017
b46a4b3
Fix test source-dependencies/ext
Duhemm Oct 20, 2017
9281d1c
Fix test source-dependencies/less-inter-inv
Duhemm Oct 20, 2017
ff34abb
Fix test sbt-dotty/example-project
Duhemm Oct 20, 2017
805eceb
Fix test compilerReporter/simple
Duhemm Oct 20, 2017
4475e9e
Fix test source-dependencies/value-class-underlying
Duhemm Oct 23, 2017
5b5b260
Disable test source-dependencies/trait-private-var
Duhemm Oct 20, 2017
df2ac08
Always run scripted tests
Duhemm Nov 6, 2017
d818a2c
Split scripted tests in two groups
Duhemm Nov 7, 2017
2075f13
Remove `Message` and `Log`
Duhemm Nov 7, 2017
39ac06b
Upgrade to sbt 1.0.3
Duhemm Nov 7, 2017
f35894c
Change name of scala-library jar in tests
Duhemm Nov 7, 2017
ef10d7c
Address review comments
Duhemm Nov 8, 2017
80d36d5
Port sbt/zinc#444
Duhemm Nov 8, 2017
7705124
Address review comments
Duhemm Nov 9, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions .drone.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,25 @@ pipeline:
- cp -R . /tmp/3/ && cd /tmp/3/
- ./project/scripts/sbt dotty-optimised/testAll

test_sbt:
test_sbt_1of2:
group: test
image: lampepfl/dotty:2017-10-20
commands:
- cp -R . /tmp/4/ && cd /tmp/4/
- ./project/scripts/sbt sbt-dotty/scripted
when:
- ./project/scripts/sbt "sbt-dotty/scripted compilerReporter/* discovery/* sbt-dotty/* source-dependencies/*1of2"
# when:
# sbt scripted tests are slow and don't run on PRs
event: [ push, tag, deployment ]
# event: [ push, tag, deployment ]

test_sbt_2of2:
group: test
image: lampepfl/dotty:2017-10-20
commands:
- cp -R . /tmp/5/ && cd /tmp/5/
- ./project/scripts/sbt "sbt-dotty/scripted source-dependencies/*2of2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great if we did not introduce this huge regression (3x) in the time needed to run the scripted tests.
@Duhemm says it is because the bridge is recompiled for each scripted test. @smarter do you have an idea how this could be fixed?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Complain to the sbt people: sbt/sbt#3469 Maybe ask someone who has a lightbend subscription? :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, we could try copy-pasting the custom implementation of scripted used by zinc, it even has parallelism and fancy colors contributed by @jvican :P. https://github.com/sbt/zinc/blob/1.x/project/Scripted.scala https://github.com/sbt/zinc/tree/1.x/internal/zinc-scripted/src/test/scala/sbt/internal/inc

# when:
# sbt scripted tests are slow and don't run on PRs
# event: [ push, tag, deployment ]

# DOCUMENTATION:
documentation:
Expand Down
1 change: 0 additions & 1 deletion build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ val `dotty-library-optimised` = Build.`dotty-library-optimised`
val `dotty-sbt-bridge` = Build.`dotty-sbt-bridge`
val `dotty-sbt-bridge-bootstrapped` = Build.`dotty-sbt-bridge-bootstrapped`
val `dotty-language-server` = Build.`dotty-language-server`
val sjsSandbox = Build.sjsSandbox
val `dotty-bench` = Build.`dotty-bench`
val `dotty-bench-bootstrapped` = Build.`dotty-bench-bootstrapped`
val `dotty-bench-optimised` = Build.`dotty-bench-optimised`
Expand Down
112 changes: 71 additions & 41 deletions compiler/src/dotty/tools/backend/jvm/GenBCode.scala
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import java.util.Optional

import scala.reflect.ClassTag
import dotty.tools.dotc.core._
import dotty.tools.dotc.sbt.ExtractDependencies
import Periods._
import SymDenotations._
import Contexts._
Expand Down Expand Up @@ -91,15 +92,17 @@ class GenBCodePipeline(val entryPoints: List[Symbol], val int: DottyBackendInter

/* ---------------- q2 ---------------- */

case class Item2(arrivalPos: Int,
mirror: asm.tree.ClassNode,
plain: asm.tree.ClassNode,
bean: asm.tree.ClassNode,
outFolder: scala.tools.nsc.io.AbstractFile) {
case class SubItem2(classNode: asm.tree.ClassNode,
file: scala.tools.nsc.io.AbstractFile)

case class Item2(arrivalPos: Int,
mirror: SubItem2,
plain: SubItem2,
bean: SubItem2) {
def isPoison = { arrivalPos == Int.MaxValue }
}

private val poison2 = Item2(Int.MaxValue, null, null, null, null)
private val poison2 = Item2(Int.MaxValue, null, null, null)
private val q2 = new _root_.java.util.LinkedList[Item2]

/* ---------------- q3 ---------------- */
Expand All @@ -113,14 +116,14 @@ class GenBCodePipeline(val entryPoints: List[Symbol], val int: DottyBackendInter
*/
case class SubItem3(
jclassName: String,
jclassBytes: Array[Byte]
jclassBytes: Array[Byte],
jclassFile: scala.tools.nsc.io.AbstractFile
)

case class Item3(arrivalPos: Int,
mirror: SubItem3,
plain: SubItem3,
bean: SubItem3,
outFolder: scala.tools.nsc.io.AbstractFile) {
bean: SubItem3) {

def isPoison = { arrivalPos == Int.MaxValue }
}
Expand All @@ -131,7 +134,7 @@ class GenBCodePipeline(val entryPoints: List[Symbol], val int: DottyBackendInter
else 1
}
}
private val poison3 = Item3(Int.MaxValue, null, null, null, null)
private val poison3 = Item3(Int.MaxValue, null, null, null)
private val q3 = new java.util.PriorityQueue[Item3](1000, i3comparator)

/*
Expand Down Expand Up @@ -228,12 +231,52 @@ class GenBCodePipeline(val entryPoints: List[Symbol], val int: DottyBackendInter
)
} else null

// ----------- create files

val classNodes = List(mirrorC, plainC, beanC)
val classFiles = classNodes.map(cls =>
if (outF != null && cls != null) {
try {
getFileForClassfile(outF, cls.name, ".class")
} catch {
case e: FileConflictException =>
ctx.error(s"error writing ${cls.name}: ${e.getMessage}")
null
}
} else null
)

// ----------- sbt's callbacks

val fullClassName = ctx.atPhase(ctx.typerPhase) { implicit ctx =>
ExtractDependencies.extractedName(claszSymbol)
}
val isLocal = fullClassName.contains("_$")

for ((cls, clsFile) <- classNodes.zip(classFiles)) {
if (cls != null) {
if (ctx.compilerCallback != null)
ctx.compilerCallback.onClassGenerated(sourceFile, convertAbstractFile(clsFile), fullClassName)
if (ctx.sbtCallback != null) {
// ctx.sbtCallback.generatedClass(sourceFile.jfile.orElse(null), clsFile.file, fullClassName)
// TODO: Check
if (isLocal)
ctx.sbtCallback.generatedLocalClass(sourceFile.jfile.orElse(null), clsFile.file)
else {
ctx.sbtCallback.generatedNonLocalClass(sourceFile.jfile.orElse(null), clsFile.file,
cls.name, fullClassName)
}
}
}
}

// ----------- hand over to pipeline-2

val item2 =
Item2(arrivalPos,
mirrorC, plainC, beanC,
outF)
SubItem2(mirrorC, classFiles(0)),
SubItem2(plainC, classFiles(1)),
SubItem2(beanC, classFiles(2)))

q2 add item2 // at the very end of this method so that no Worker2 thread starts mutating before we're done.

Expand Down Expand Up @@ -263,12 +306,12 @@ class GenBCodePipeline(val entryPoints: List[Symbol], val int: DottyBackendInter
}
else {
try {
localOptimizations(item.plain)
localOptimizations(item.plain.classNode)
addToQ3(item)
} catch {
case ex: Throwable =>
ex.printStackTrace()
ctx.error(s"Error while emitting ${item.plain.name}\n${ex.getMessage}")
ctx.error(s"Error while emitting ${item.plain.classNode.name}\n${ex.getMessage}")
}
}
}
Expand All @@ -282,19 +325,22 @@ class GenBCodePipeline(val entryPoints: List[Symbol], val int: DottyBackendInter
cw.toByteArray
}

val Item2(arrivalPos, mirror, plain, bean, outFolder) = item
val Item2(arrivalPos,
SubItem2(mirror, mirrorFile),
SubItem2(plain, plainFile),
SubItem2(bean, beanFile)) = item

val mirrorC = if (mirror == null) null else SubItem3(mirror.name, getByteArray(mirror))
val plainC = SubItem3(plain.name, getByteArray(plain))
val beanC = if (bean == null) null else SubItem3(bean.name, getByteArray(bean))
val mirrorC = if (mirror == null) null else SubItem3(mirror.name, getByteArray(mirror), mirrorFile)
val plainC = SubItem3(plain.name, getByteArray(plain), plainFile)
val beanC = if (bean == null) null else SubItem3(bean.name, getByteArray(bean), beanFile)

if (AsmUtils.traceSerializedClassEnabled && plain.name.contains(AsmUtils.traceSerializedClassPattern)) {
if (mirrorC != null) AsmUtils.traceClass(mirrorC.jclassBytes)
AsmUtils.traceClass(plainC.jclassBytes)
if (beanC != null) AsmUtils.traceClass(beanC.jclassBytes)
}

q3 add Item3(arrivalPos, mirrorC, plainC, beanC, outFolder)
q3 add Item3(arrivalPos, mirrorC, plainC, beanC)

}

Expand Down Expand Up @@ -394,25 +440,10 @@ class GenBCodePipeline(val entryPoints: List[Symbol], val int: DottyBackendInter
/* Pipeline that writes classfile representations to disk. */
private def drainQ3() = {

def sendToDisk(cfr: SubItem3, outFolder: scala.tools.nsc.io.AbstractFile): Unit = {
def sendToDisk(cfr: SubItem3): Unit = {
if (cfr != null){
val SubItem3(jclassName, jclassBytes) = cfr
try {
val outFile =
if (outFolder == null) null
else getFileForClassfile(outFolder, jclassName, ".class")
bytecodeWriter.writeClass(jclassName, jclassName, jclassBytes, outFile)

val className = jclassName.replace('/', '.')
if (ctx.compilerCallback != null)
ctx.compilerCallback.onClassGenerated(sourceFile, convertAbstractFile(outFile), className)
if (ctx.sbtCallback != null)
ctx.sbtCallback.generatedClass(sourceFile.jfile.orElse(null), outFile.file, className)
}
catch {
case e: FileConflictException =>
ctx.error(s"error writing $jclassName: ${e.getMessage}")
}
val SubItem3(jclassName, jclassBytes, jclassFile) = cfr
bytecodeWriter.writeClass(jclassName, jclassName, jclassBytes, jclassFile)
}
}

Expand All @@ -425,10 +456,9 @@ class GenBCodePipeline(val entryPoints: List[Symbol], val int: DottyBackendInter
moreComing = !incoming.isPoison
if (moreComing) {
val item = incoming
val outFolder = item.outFolder
sendToDisk(item.mirror, outFolder)
sendToDisk(item.plain, outFolder)
sendToDisk(item.bean, outFolder)
sendToDisk(item.mirror)
sendToDisk(item.plain)
sendToDisk(item.bean)
expected += 1
}
}
Expand Down
3 changes: 2 additions & 1 deletion compiler/src/dotty/tools/dotc/core/Names.scala
Original file line number Diff line number Diff line change
Expand Up @@ -424,7 +424,8 @@ object Names {
"dotty$tools$dotc$core$NameOps$NameDecorator$$functionArityFor$extension",
"dotty$tools$dotc$typer$Checking$CheckNonCyclicMap$$apply",
"$plus$plus",
"readConstant")
"readConstant",
"extractedName")
.contains(elem.getMethodName))
}

Expand Down
14 changes: 13 additions & 1 deletion compiler/src/dotty/tools/dotc/core/SymDenotations.scala
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import collection.BitSet
import dotty.tools.io.AbstractFile
import Decorators.SymbolIteratorDecorator
import ast._
import ast.Trees._
import annotation.tailrec
import CheckRealizable._
import util.SimpleIdentityMap
Expand Down Expand Up @@ -334,6 +335,14 @@ object SymDenotations {
case Nil => Nil
}

final def children(implicit ctx: Context): List[Symbol] =
this.annotations.filter(_.symbol == ctx.definitions.ChildAnnot).map { annot =>
// refer to definition of Annotation.makeChild
annot.tree match {
case Apply(TypeApply(_, List(tpTree)), _) => tpTree.symbol
}
}

/** The denotation is completed: info is not a lazy type and attributes have defined values */
final def isCompleted: Boolean = !myInfo.isInstanceOf[LazyType]

Expand Down Expand Up @@ -919,13 +928,16 @@ object SymDenotations {
*/
final def topLevelClass(implicit ctx: Context): Symbol = {
def topLevel(d: SymDenotation): Symbol = {
if (d.isEffectiveRoot || (d is PackageClass) || (d.owner is PackageClass)) d.symbol
if (d.isTopLevelClass) d.symbol
else topLevel(d.owner)
}
val sym = topLevel(this)
if (sym.isClass) sym else sym.moduleClass
}

final def isTopLevelClass(implicit ctx: Context): Boolean =
this.isEffectiveRoot || (this is PackageClass) || (this.owner is PackageClass)

/** The package class containing this denotation */
final def enclosingPackageClass(implicit ctx: Context): Symbol =
if (this is PackageClass) symbol else owner.enclosingPackageClass
Expand Down
Loading