Skip to content

Too many open files when running dotty.tools.dotc.IdempotencyTest.idempotency. #3450

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Duhemm opened this issue Nov 9, 2017 · 5 comments
Closed

Too many open files when running dotty.tools.dotc.IdempotencyTest.idempotency. #3450

Duhemm opened this issue Nov 9, 2017 · 5 comments
Labels
area:infrastructure area:testing itype:bug

Comments

@Duhemm
Copy link
Contributor

Duhemm commented Nov 9, 2017

On my machine, running the test dotty.tools.dotc.IdempotencyTests.idempotency, because there are too many files open.

> testOnly dotty.tools.dotc.IdempotencyTests -- --tests=idempotency

First bad commit: 92a25cb
ref #3395

Partial stack trace 
[info] Test dotty.tools.dotc.IdempotencyTests.idempotency started
[=================================>     ] completed (1553/1776, 71s)
error while loading Short,
 /Library/Java/Extensions: Too many open files

error while loading Integer,
 /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/ext: Too many open files

error while loading Throwable,
 /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/ext: Too many open files
java.nio.file.FileSystemException: /Users/martin/Documents/Projects.nosync/Duhemm/dotty/compiler/../out/posIdempotency2/pos/t2503/SB.tasty: Too many open files
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.Files.newByteChannel(Files.java:361)
        at java.nio.file.Files.createFile(Files.java:632)
        at dotty.tools.io.AbstractFile.fileOrSubdirectoryNamed(AbstractFile.scala:247)
        at dotty.tools.io.AbstractFile.fileNamed(AbstractFile.scala:258)
        at scala.tools.nsc.backend.jvm.BytecodeWriters.getFile(BytecodeWriters.scala:35)
        at scala.tools.nsc.backend.jvm.BytecodeWriters.getFile$(BytecodeWriters.scala:28)
        at dotty.tools.backend.jvm.GenBCodePipeline.getFile(GenBCode.scala:59)
        at scala.tools.nsc.backend.jvm.BCodeHelpers.getFileForClassfile(BCodeHelpers.scala:38)
        at scala.tools.nsc.backend.jvm.BCodeHelpers.getFileForClassfile$(BCodeHelpers.scala:37)
        at dotty.tools.backend.jvm.GenBCodePipeline.getFileForClassfile(GenBCode.scala:59)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.$anonfun$visit$2(GenBCode.scala:209)
        at scala.Option.foreach(Option.scala:257)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.visit(GenBCode.scala:204)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.run(GenBCode.scala:152)
        at dotty.tools.backend.jvm.GenBCodePipeline.buildAndSendToDisk(GenBCode.scala:367)
        at dotty.tools.backend.jvm.GenBCodePipeline.run(GenBCode.scala:333)
        at dotty.tools.backend.jvm.GenBCode.run(GenBCode.scala:54)
        at dotty.tools.dotc.core.Phases$Phase.$anonfun$runOn$1(Phases.scala:289)
        at scala.collection.immutable.List.map(List.scala:283)
        at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:287)
        at dotty.tools.dotc.core.Phases$Phase.runOn$(Phases.scala:286)
        at dotty.tools.backend.jvm.GenBCode.runOn(GenBCode.scala:38)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$3(Run.scala:125)

error while loading Long,
 /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/ext: Too many open files
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at dotty.tools.dotc.util.Stats$.trackTime(Stats.scala:47)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$2(Run.scala:123)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$2$adapted(Run.scala:121)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:191)
        at dotty.tools.dotc.Run.runPhases$1(Run.scala:121)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$1(Run.scala:141)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:86)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:107)
        at dotty.tools.dotc.Run.compileSources(Run.scala:94)
        at dotty.tools.dotc.Run.compile(Run.scala:78)
        at dotty.tools.dotc.Driver.doCompile(Driver.scala:26)
        at dotty.tools.dotc.Driver.process(Driver.scala:124)
        at dotty.tools.dotc.Driver.process(Driver.scala:93)
        at dotty.tools.vulpix.ParallelTesting$Test.compile(ParallelTesting.scala:386)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.$anonfun$checkTestSource$1(ParallelTesting.scala:451)
        at dotty.tools.vulpix.ParallelTesting$Test.tryCompile(ParallelTesting.scala:317)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.checkTestSource(ParallelTesting.scala:449)
        at dotty.tools.vulpix.ParallelTesting$Test$LoggedRunnable.run(ParallelTesting.scala:204)
        at dotty.tools.vulpix.ParallelTesting$Test$LoggedRunnable.run$(ParallelTesting.scala:203)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.run(ParallelTesting.scala:447)
        at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
java.nio.file.FileSystemException: /Users/martin/Documents/Projects.nosync/Duhemm/dotty/compiler/../out/posIdempotency2/pos/t2503/Test.tasty: Too many open files
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
        at java.nio.file.Files.newByteChannel(Files.java:361)
        at java.nio.file.Files.createFile(Files.java:632)
        at dotty.tools.io.AbstractFile.fileOrSubdirectoryNamed(AbstractFile.scala:247)
        at dotty.tools.io.AbstractFile.fileNamed(AbstractFile.scala:258)
        at scala.tools.nsc.backend.jvm.BytecodeWriters.getFile(BytecodeWriters.scala:35)
        at scala.tools.nsc.backend.jvm.BytecodeWriters.getFile$(BytecodeWriters.scala:28)
        at dotty.tools.backend.jvm.GenBCodePipeline.getFile(GenBCode.scala:59)
        at scala.tools.nsc.backend.jvm.BCodeHelpers.getFileForClassfile(BCodeHelpers.scala:38)
        at scala.tools.nsc.backend.jvm.BCodeHelpers.getFileForClassfile$(BCodeHelpers.scala:37)
        at dotty.tools.backend.jvm.GenBCodePipeline.getFileForClassfile(GenBCode.scala:59)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.$anonfun$visit$2(GenBCode.scala:209)
        at scala.Option.foreach(Option.scala:257)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.visit(GenBCode.scala:204)
        at dotty.tools.backend.jvm.GenBCodePipeline$Worker1.run(GenBCode.scala:152)
        at dotty.tools.backend.jvm.GenBCodePipeline.buildAndSendToDisk(GenBCode.scala:367)
        at dotty.tools.backend.jvm.GenBCodePipeline.run(GenBCode.scala:333)
        at dotty.tools.backend.jvm.GenBCode.run(GenBCode.scala:54)
        at dotty.tools.dotc.core.Phases$Phase.$anonfun$runOn$1(Phases.scala:289)
        at scala.collection.immutable.List.map(List.scala:283)
        at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:287)
        at dotty.tools.dotc.core.Phases$Phase.runOn$(Phases.scala:286)
        at dotty.tools.backend.jvm.GenBCode.runOn(GenBCode.scala:38)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$3(Run.scala:125)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at dotty.tools.dotc.util.Stats$.trackTime(Stats.scala:47)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$2(Run.scala:123)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$2$adapted(Run.scala:121)
        at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
        at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:191)
        at dotty.tools.dotc.Run.runPhases$1(Run.scala:121)
        at dotty.tools.dotc.Run.$anonfun$compileUnits$1(Run.scala:141)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:86)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:107)
        at dotty.tools.dotc.Run.compileSources(Run.scala:94)
        at dotty.tools.dotc.Run.compile(Run.scala:78)
        at dotty.tools.dotc.Driver.doCompile(Driver.scala:26)
        at dotty.tools.dotc.Driver.process(Driver.scala:124)
        at dotty.tools.dotc.Driver.process(Driver.scala:93)
        at dotty.tools.vulpix.ParallelTesting$Test.compile(ParallelTesting.scala:386)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.$anonfun$checkTestSource$1(ParallelTesting.scala:451)
        at dotty.tools.vulpix.ParallelTesting$Test.tryCompile(ParallelTesting.scala:317)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.checkTestSource(ParallelTesting.scala:449)
        at dotty.tools.vulpix.ParallelTesting$Test$LoggedRunnable.run(ParallelTesting.scala:204)
        at dotty.tools.vulpix.ParallelTesting$Test$LoggedRunnable.run$(ParallelTesting.scala:203)
        at dotty.tools.vulpix.ParallelTesting$PosTest$$anon$2.run(ParallelTesting.scala:447)
        at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

Error while emitting t2503.scala
/Users/martin/Documents/Projects.nosync/Duhemm/dotty/compiler/../out/posIdempotency2/pos/t2503/SB.tasty: Too many open files

Error while emitting t2503.scala
/Users/martin/Documents/Projects.nosync/Duhemm/dotty/compiler/../out/posIdempotency2/pos/t2503/Test.tasty: Too many open files

error while loading Throwable,
 /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/ext: Too many open files

error while loading Byte,
 /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre/lib/ext: Too many open files

(And many more)
@allanrenucci
Copy link
Contributor

@AleksanderBG you've encountered something similar. What OS are you running?

@abgruszecki
Copy link
Contributor

@allanrenucci - macOS. In my case, the error appeared after running the entire test suite multiple times in quick succession, so that may have been the cause.

@nicolasstucki
Copy link
Contributor

I have encountered the same issue on Mac OS :(

@odersky
Copy link
Contributor

odersky commented Nov 11, 2017

Same for me on MacOS. I can't even run a single test anymore. It runs out of files in compile/pos. Could it be that some file handles are not closed anymore?

@nicolasstucki
Copy link
Contributor

By bisecting again I found 7ff1cb6 to be the first bad commit.

odersky added a commit that referenced this issue Nov 12, 2017
@odersky
Merge pull request #3458 from dotty-staging/fix-#3450
03d660a 
Fix #3450: Partial revert of 7ff1cb6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:infrastructure area:testing itype:bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@odersky @Duhemm @nicolasstucki @allanrenucci @abgruszecki