[GR-38994] Add support for -XX:HeapDumpOnOutOfMemoryError.

docs/reference-manual/native-image/guides/create-heap-dump-from-native-executable.md

@@ -7,14 +7,16 @@ permalink: /reference-manual/native-image/guides/create-heap-dump/
 
 # Create a Heap Dump from a Native Executable
 
-You can create a heap dump of a running executable to monitor its execution. Just like any other Java heap dump, it can be opened with the [VisualVM](../../../tools/visualvm.md) tool.
+You can create a heap dump of a running executable to monitor its execution.
+Just like any other Java heap dump, it can be opened with the [VisualVM](../../../tools/visualvm.md) tool.
 
-To enable heap dump support, native executables must be built with the `--enable-monitoring=heapdump` option. Heap dumps can then be created in three different ways:
+To enable heap dump support, native executables must be built with the `--enable-monitoring=heapdump` option. Heap dumps can then be created in different ways:
 
 1. Create heap dumps with VisualVM.
-2. Dump the initial heap of a native executable using the `-XX:+DumpHeapAndExit` command-line option.
-3. Create heap dumps sending a `SIGUSR1` signal at run time.
-4. Create heap dumps programmatically using the [`org.graalvm.nativeimage.VMRuntime#dumpHeap`](https://github.com/oracle/graal/blob/master/substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/VMInspectionOptions.java) API.
+2. The command-line option `-XX:+HeapDumpOnOutOfMemoryError` can be used to create a heap dump when the native executable runs out of Java heap memory.
+3. Dump the initial heap of a native executable using the `-XX:+DumpHeapAndExit` command-line option.
+4. Create heap dumps sending a `SIGUSR1` signal at run time.
+5. Create heap dumps programmatically using the [`org.graalvm.nativeimage.VMRuntime#dumpHeap`](https://github.com/oracle/graal/blob/master/substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/VMInspectionOptions.java) API.
 
 All approaches are described below.
 
@@ -30,6 +32,19 @@ For this, you need to add `jvmstat` to the `--enable-monitoring` option (for exa
 This will allow VisualVM to pick up and list running Native Image processes.
 You can then request heap dumps in the same way you can request them when your application runs on the JVM (for example, right-click on the process, then select "Heap Dump").
 
+## Create a Heap Dump on `OutOfMemoryError`
+
+Start the application with the option `-XX:+HeapDumpOnOutOfMemoryError` to get a heap dump when the native executable throws an `OutOfMemoryError` because it ran out of Java heap memory.
+The heap dump is created in a file named `svm-heapdump-<PID>-OOME.hprof`.
+For example:
+
+```shell
+./mem-leak-example -XX:+HeapDumpOnOutOfMemoryError
+Dumping heap to svm-heapdump-67799-OOME.hprof ...
+Heap dump file created [10046752 bytes in 0.49 secs]
+Exception in thread "main" java.lang.OutOfMemoryError: Garbage-collected heap size exceeded.
+```
+
 ## Dump the Initial Heap of a Native Executable
 
 Use the `-XX:+DumpHeapAndExit` command-line option to dump the initial heap of a native executable.
@@ -95,6 +110,7 @@ For other installation options, visit the [Downloads section](https://www.graalv
             for (int i = 0; i < 1000; i++) {
                 CROWD.add(new Person());
             }
+
             long pid = ProcessProperties.getProcessID();
             StringBuffer sb1 = new StringBuffer(100);
             sb1.append(DATE_FORMATTER.format(new Date()));
@@ -105,6 +121,7 @@ For other installation options, visit the [Downloads section](https://www.graalv
             sb1.append("to dump the heap into the working directory.\n");
             sb1.append("Starting thread!");
             System.out.println(sb1);
+
             SVMHeapDump t = new SVMHeapDump();
             t.start();
             while (t.isAlive()) {
@@ -120,17 +137,17 @@ For other installation options, visit the [Downloads section](https://www.graalv
     }
 
     class Person {
-            private static Random R = new Random();
-            private String name;
-            private int age;
+        private static Random R = new Random();
+        private String name;
+        private int age;
 
-            public Person() {
-                byte[] array = new byte[7];
-                R.nextBytes(array);
-                name = new String(array, Charset.forName("UTF-8"));
-                age = R.nextInt(100);
-            }
+        public Person() {
+            byte[] array = new byte[7];
+            R.nextBytes(array);
+            name = new String(array, Charset.forName("UTF-8"));
+            age = R.nextInt(100);
         }
+    }
     ```
 
 3. Build a native executable:

substratevm/CHANGELOG.md

@@ -10,6 +10,7 @@ This changelog summarizes major changes to GraalVM Native Image.
 * (GR-39406) All classes can now be used at image build time, even when they are not explicitly configured as `--initialize-at-build-time`. Note, however, that still only classes configured as `--initialize-at-build-time` are allowed in the image heap.
 * (GR-46392) Add `--parallelism` option to control how many threads are used by the build process.
 * (GR-46392) Add build resources section to the build output that shows the memory and thread limits of the build process.
+* (GR-38994) Together with Red Hat, we added support for `-XX:+HeapDumpOnOutOfMemoryError`.
 * (GR-47365) Throw `MissingReflectionRegistrationError` when attempting to create a proxy class without having it registered at build-time, instead of a `VMError`.
 
 ## Version 23.0.0

substratevm/src/com.oracle.svm.core.posix/src/com/oracle/svm/core/posix/PosixRawFileOperationSupport.java

@@ -30,7 +30,9 @@
 import org.graalvm.nativeimage.ImageSingletons;
 import org.graalvm.nativeimage.Platform;
 import org.graalvm.nativeimage.Platforms;
+import org.graalvm.nativeimage.c.type.CCharPointer;
 import org.graalvm.nativeimage.c.type.CTypeConversion;
+import org.graalvm.nativeimage.impl.UnmanagedMemorySupport;
 import org.graalvm.word.Pointer;
 import org.graalvm.word.SignedWord;
 import org.graalvm.word.UnsignedWord;
@@ -53,27 +55,61 @@ public PosixRawFileOperationSupport(boolean useNativeByteOrder) {
         super(useNativeByteOrder);
     }
 
+    @Override
+    public CCharPointer allocateCPath(String path) {
+        byte[] data = path.getBytes();
+        CCharPointer filename = ImageSingletons.lookup(UnmanagedMemorySupport.class).malloc(WordFactory.unsigned(data.length + 1));
+        if (filename.isNull()) {
+            return WordFactory.nullPointer();
+        }
+
+        for (int i = 0; i < data.length; i++) {
+            filename.write(i, data[i]);
+        }
+        filename.write(data.length, (byte) 0);
+        return filename;
+    }
+
     @Override
     public RawFileDescriptor create(File file, FileCreationMode creationMode, FileAccessMode accessMode) {
         String path = file.getPath();
         int flags = parseMode(creationMode) | parseMode(accessMode);
         return open0(path, flags);
     }
 
+    @Override
+    @Uninterruptible(reason = "Called from uninterruptible code.", mayBeInlined = true)
+    public RawFileDescriptor create(CCharPointer cPath, FileCreationMode creationMode, FileAccessMode accessMode) {
+        int flags = parseMode(creationMode) | parseMode(accessMode);
+        return open0(cPath, flags);
+    }
+
     @Override
     public RawFileDescriptor open(File file, FileAccessMode mode) {
         String path = file.getPath();
         int flags = parseMode(mode);
         return open0(path, flags);
     }
 
+    @Override
+    @Uninterruptible(reason = "Called from uninterruptible code.", mayBeInlined = true)
+    public RawFileDescriptor open(CCharPointer cPath, FileAccessMode mode) {
+        int flags = parseMode(mode);
+        return open0(cPath, flags);
+    }
+
     private static RawFileDescriptor open0(String path, int flags) {
-        int permissions = PosixStat.S_IRUSR() | PosixStat.S_IWUSR();
         try (CTypeConversion.CCharPointerHolder cPath = CTypeConversion.toCString(path)) {
-            return WordFactory.signed(Fcntl.NoTransitions.open(cPath.get(), flags, permissions));
+            return open0(cPath.get(), flags);
         }
     }
 
+    @Uninterruptible(reason = "Called from uninterruptible code.", mayBeInlined = true)
+    private static RawFileDescriptor open0(CCharPointer cPath, int flags) {
+        int permissions = PosixStat.S_IRUSR() | PosixStat.S_IWUSR();
+        return WordFactory.signed(Fcntl.NoTransitions.open(cPath, flags, permissions));
+    }
+
     @Uninterruptible(reason = "Called from uninterruptible code.", mayBeInlined = true)
     @Override
     public boolean isValid(RawFileDescriptor fd) {
@@ -154,6 +190,7 @@ private static int getPosixFileDescriptor(RawFileDescriptor fd) {
         return result;
     }
 
+    @Uninterruptible(reason = "Called from uninterruptible code.", mayBeInlined = true)
     private static int parseMode(FileCreationMode mode) {
         switch (mode) {
             case CREATE:
@@ -165,6 +202,7 @@ private static int parseMode(FileCreationMode mode) {
         }
     }
 
+    @Uninterruptible(reason = "Called from uninterruptible code.", mayBeInlined = true)
     private static int parseMode(FileAccessMode mode) {
         switch (mode) {
             case READ:

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/DumpHeapOnSignalFeature.java

@@ -25,16 +25,10 @@
 package com.oracle.svm.core;
 
 import java.io.IOException;
-import java.text.DateFormat;
-import java.text.SimpleDateFormat;
-import java.util.Date;
-import java.util.TimeZone;
-
-import org.graalvm.nativeimage.ProcessProperties;
-import org.graalvm.nativeimage.VMRuntime;
 
 import com.oracle.svm.core.feature.AutomaticallyRegisteredFeature;
 import com.oracle.svm.core.feature.InternalFeature;
+import com.oracle.svm.core.heap.dump.HeapDumping;
 import com.oracle.svm.core.jdk.RuntimeSupport;
 import com.oracle.svm.core.log.Log;
 
@@ -50,7 +44,8 @@ public boolean isInConfiguration(IsInConfigurationAccess access) {
 
     @Override
     public void beforeAnalysis(BeforeAnalysisAccess access) {
-        RuntimeSupport.getRuntimeSupport().addStartupHook(new DumpHeapStartupHook());
+        RuntimeSupport.getRuntimeSupport().addInitializationHook(new DumpHeapStartupHook());
+        RuntimeSupport.getRuntimeSupport().addTearDownHook(new DumpHeapTeardownHook());
     }
 }
 
@@ -60,24 +55,30 @@ public void execute(boolean isFirstIsolate) {
         if (isFirstIsolate && SubstrateOptions.EnableSignalHandling.getValue()) {
             DumpHeapReport.install();
         }
+
+        if (SubstrateOptions.HeapDumpOnOutOfMemoryError.getValue()) {
+            HeapDumping.singleton().initializeDumpHeapOnOutOfMemoryError();
+        }
     }
 }
 
-class DumpHeapReport implements Signal.Handler {
-    private static final TimeZone UTC_TIMEZONE = TimeZone.getTimeZone("UTC");
+final class DumpHeapTeardownHook implements RuntimeSupport.Hook {
+    @Override
+    public void execute(boolean isFirstIsolate) {
+        /* Do this unconditionally, the runtime option could have changed in the meanwhile. */
+        HeapDumping.singleton().teardownDumpHeapOnOutOfMemoryError();
+    }
+}
 
+class DumpHeapReport implements Signal.Handler {
     static void install() {
         Signal.handle(new Signal("USR1"), new DumpHeapReport());
     }
 
     @Override
     public void handle(Signal arg0) {
-        DateFormat dateFormat = new SimpleDateFormat("yyyyMMdd'T'HHmmss'Z'");
-        dateFormat.setTimeZone(UTC_TIMEZONE);
-        String defaultHeapDumpFileName = "svm-heapdump-" + ProcessProperties.getProcessID() + "-" + dateFormat.format(new Date()) + ".hprof";
-        String heapDumpPath = SubstrateOptions.getHeapDumpPath(defaultHeapDumpFileName);
         try {
-            VMRuntime.dumpHeap(heapDumpPath, true);
+            HeapDumping.singleton().dumpHeap(true);
         } catch (IOException e) {
             Log.log().string("IOException during dumpHeap: ").string(e.getMessage()).newline();
         }

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/SubstrateOptions.java

@@ -31,7 +31,6 @@
 import static org.graalvm.compiler.options.OptionType.Expert;
 import static org.graalvm.compiler.options.OptionType.User;
 
-import java.nio.file.Files;
 import java.nio.file.InvalidPathException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
@@ -825,21 +824,14 @@ public Boolean getValue(OptionValues values) {
         }
     };
 
+    @Option(help = "Dump heap to file (see HeapDumpPath) when the executable throws a java.lang.OutOfMemoryError because it ran out of Java heap.")//
+    public static final RuntimeOptionKey<Boolean> HeapDumpOnOutOfMemoryError = new RuntimeOptionKey<>(false);
+
     @Option(help = "The path (filename or directory) where heap dumps are created (defaults to the working directory).")//
     public static final RuntimeOptionKey<String> HeapDumpPath = new RuntimeOptionKey<>("", Immutable);
 
-    /* Utility method that follows the `-XX:HeapDumpPath` behavior of the JVM. */
-    public static String getHeapDumpPath(String defaultFilename) {
-        String heapDumpFilenameOrDirectory = HeapDumpPath.getValue();
-        if (heapDumpFilenameOrDirectory.isEmpty()) {
-            return defaultFilename;
-        }
-        var targetPath = Paths.get(heapDumpFilenameOrDirectory);
-        if (Files.isDirectory(targetPath)) {
-            targetPath = targetPath.resolve(defaultFilename);
-        }
-        return targetPath.toFile().getAbsolutePath();
-    }
+    @Option(help = "A prefix that is used for heap dump filenames if no heap dump filename was specified explicitly.")//
+    public static final HostedOptionKey<String> HeapDumpDefaultFilenamePrefix = new HostedOptionKey<>("svm-heapdump-");
 
     @Option(help = "Create a heap dump and exit.")//
     public static final RuntimeOptionKey<Boolean> DumpHeapAndExit = new RuntimeOptionKey<>(false, Immutable);

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/UnmanagedMemoryUtil.java

@@ -51,7 +51,7 @@
  * </ul>
  * <p>
  * In some situations (e.g., during a serial GC or if it is guaranteed that all involved objects are
- * not yet visible to other threads), the methods in this class may also be used for objects the
+ * not yet visible to other threads), the methods in this class may also be used for objects that
  * live in the Java heap. However, those usages should be kept to a minimum.
  */
 public final class UnmanagedMemoryUtil {

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/VMInspectionOptions.java

@@ -37,8 +37,8 @@
 import org.graalvm.nativeimage.Platform;
 import org.graalvm.nativeimage.Platform.WINDOWS;
 import org.graalvm.nativeimage.Platforms;
-import org.graalvm.nativeimage.VMRuntime;
 
+import com.oracle.svm.core.heap.dump.HeapDumping;
 import com.oracle.svm.core.jdk.management.ManagementAgentModule;
 import com.oracle.svm.core.option.APIOption;
 import com.oracle.svm.core.option.HostedOptionKey;
@@ -109,9 +109,9 @@ public static boolean hasHeapDumpSupport() {
 
     public static boolean dumpImageHeap() {
         if (hasHeapDumpSupport()) {
-            String absoluteHeapDumpPath = SubstrateOptions.getHeapDumpPath(SubstrateOptions.Name.getValue() + ".hprof");
+            String absoluteHeapDumpPath = HeapDumping.getHeapDumpPath(SubstrateOptions.Name.getValue() + ".hprof");
             try {
-                VMRuntime.dumpHeap(absoluteHeapDumpPath, true);
+                HeapDumping.singleton().dumpHeap(absoluteHeapDumpPath, true);
             } catch (IOException e) {
                 System.err.println("Failed to create heap dump:");
                 e.printStackTrace();

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/heap/OutOfMemoryUtil.java

@@ -25,14 +25,19 @@
 package com.oracle.svm.core.heap;
 
 import com.oracle.svm.core.SubstrateGCOptions;
+import com.oracle.svm.core.SubstrateOptions;
 import com.oracle.svm.core.Uninterruptible;
+import com.oracle.svm.core.VMInspectionOptions;
 import com.oracle.svm.core.headers.LibC;
+import com.oracle.svm.core.heap.dump.HeapDumping;
 import com.oracle.svm.core.jdk.JDKUtils;
+import com.oracle.svm.core.jdk.UninterruptibleUtils.AtomicBoolean;
 import com.oracle.svm.core.log.Log;
 import com.oracle.svm.core.util.VMError;
 
 public class OutOfMemoryUtil {
     private static final OutOfMemoryError OUT_OF_MEMORY_ERROR = new OutOfMemoryError("Garbage-collected heap size exceeded.");
+    private static final AtomicBoolean HEAP_DUMPED = new AtomicBoolean(false);
 
     @RestrictHeapAccess(access = RestrictHeapAccess.Access.NO_ALLOCATION, reason = "Can't allocate when out of memory.")
     public static OutOfMemoryError heapSizeExceeded() {
@@ -42,6 +47,10 @@ public static OutOfMemoryError heapSizeExceeded() {
     @Uninterruptible(reason = "Not uninterruptible but it doesn't matter for the callers.", calleeMustBe = false)
     @RestrictHeapAccess(access = RestrictHeapAccess.Access.NO_ALLOCATION, reason = "Can't allocate while out of memory.")
     public static OutOfMemoryError reportOutOfMemoryError(OutOfMemoryError error) {
+        if (VMInspectionOptions.hasHeapDumpSupport() && SubstrateOptions.HeapDumpOnOutOfMemoryError.getValue() && HEAP_DUMPED.compareAndSet(false, true)) {
+            HeapDumping.singleton().dumpHeapOnOutOfMemoryError();
+        }
+
         if (SubstrateGCOptions.ExitOnOutOfMemoryError.getValue()) {
             if (LibC.isSupported()) {
                 Log.log().string("Terminating due to java.lang.OutOfMemoryError: ").string(JDKUtils.getRawMessage(error)).newline();