JVM crash with "There is insufficient memory for the Java Runtime Environment to continue"

[retronym]

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 123207680 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:

Seen by @soc, @odersky and I intermittently in the past two days.

[retronym]

Example: https://scala-ci.typesafe.com/job/dotty-master-validate-partest-bootstrapped/953/

[retronym]

We think this is due to an increase in the memory usage by the dotty build. @adriaanm planned to reduce the job limit on each Jenkins worker (the EC2 instances have 16GB ram).

[retronym]

Partest currently uses _refArrayOps. We should remove the _ prefixed versions and make the original names implicit again with the new return types, as suggested in the comment. Let's do that in a separate PR, though, so we can clearly see why we're doing a bootstrap.

[SethTisue]

@adriaanm, at least, has still seen PR validation failure(s?) in the last few days, even after the number of executors per behemoth was reduced in 2a83d37 — including a strange EOF error, perhaps from a forked JVM that died and we don't have the right error reporting in place?

[SethTisue]

https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-bootstrap/ has failed several nights in a row now with this error. the failures started when we bumped STARR from M5 to RC1 in scala/scala@7507765 but that's probably a coincidence.

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000a0580000, 233832448, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 233832448 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/jenkins/workspace/scala-2.12.x-integrate-bootstrap/hs_err_pid26131.log
Exception in thread "Thread-33" java.io.EOFException
    at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2626)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1321)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
    at sbt.React.react(ForkTests.scala:114)
    at sbt.ForkTests$$anonfun$mainTestTask$1$Acceptor$2$.run(ForkTests.scala:74)
    at java.lang.Thread.run(Thread.java:745)

[adriaanm]

Could also be because we started building 2.12.0 and 2.12.x simultaneously?
Didn't check parallelism setting tho
On Sun, Sep 11, 2016 at 21:39 Seth Tisue notifications@github.com wrote:

https://scala-ci.typesafe.com/job/scala-2.12.x-integrate-bootstrap/ has
failed several nights in a row now with this error. the failures started
when we bumped STARR from M5 to RC1 in scala/scala@7507765
scala/scala@7507765
but that's probably a coincidence.

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000a0580000, 233832448, 0) failed; error='Cannot allocate memory' (errno=12)

There is insufficient memory for the Java Runtime Environment to continue.

Native memory allocation (mmap) failed to map 233832448 bytes for committing reserved memory.

An error report file with more information is saved as:

/home/jenkins/workspace/scala-2.12.x-integrate-bootstrap/hs_err_pid26131.log

Exception in thread "Thread-33" java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2626)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1321)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
at sbt.React.react(ForkTests.scala:114)
at sbt.ForkTests$$anonfun$mainTestTask$1$Acceptor$2$.run(ForkTests.scala:74)
at java.lang.Thread.run(Thread.java:745)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#181 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAFjy0RI0_qWK6Rqk5zGOrfnkRgZeNGLks5qpFj9gaJpZM4JXS3d
.

[SethTisue]

Could also be because we started building 2.12.0 and 2.12.x simultaneously

I think that's likely.

lately, we have not seen the memory crash causing spurious PR validation failures.

but most scala-2.12.0-integrate-bootstrap and scala-2.12.x-integrate-bootstrap runs have been failing. the bootstrap jobs run on jenkins-worker-ubuntu-publish, not on the worker behemoths.

does 2a83d37 only affect the behemoths?