OutOfMemoryError reported by jenkins validation build after org.eclipse.ui.tests.UiTestSuite

[iloveeclipse]

See https://ci.eclipse.org/platform/job/eclipse.platform.ui/job/PR-2431/1/consoleFull that didn't changed code, just updated bundle versions, so the problem must be coming from either environment or code changes before.

The question is: what is causing that, and which heap space is actually used on jenkins?

Exception in thread "Active Thread: Equinox Container: aa943991-8bd5-44f7-a55a-1a7bc7647820" java.lang.OutOfMemoryError: Java heap space
Exception in thread "WorkbenchTestable" org.eclipse.swt.SWTException: Failed to execute runnable (java.lang.OutOfMemoryError: Java heap space)
	at org.eclipse.swt.SWT.error(SWT.java:4922)
	at org.eclipse.swt.SWT.error(SWT.java:4837)
	at org.eclipse.swt.widgets.Synchronizer.syncExec(Synchronizer.java:209)
	at org.eclipse.ui.internal.UISynchronizer.syncExec(UISynchronizer.java:133)
	at org.eclipse.swt.widgets.Display.syncExec(Unknown Source)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.runTest(E4Testable.java:118)
	at org.eclipse.tycho.surefire.osgibooter.AbstractUITestApplication.runTests(AbstractUITestApplication.java:38)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.lambda$0(E4Testable.java:79)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-11"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-22"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-23"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-24"

[iloveeclipse]

Also in https://ci.eclipse.org/platform/job/eclipse.platform.ui/view/change-requests/job/PR-2224/11/consoleFull

Exception in thread "WorkbenchTestable" org.eclipse.swt.SWTException: Failed to execute runnable (java.lang.OutOfMemoryError: Java heap space)
	at org.eclipse.swt.SWT.error(SWT.java:4922)
	at org.eclipse.swt.SWT.error(SWT.java:4837)
	at org.eclipse.swt.widgets.Synchronizer.syncExec(Synchronizer.java:209)
	at org.eclipse.ui.internal.UISynchronizer.syncExec(UISynchronizer.java:133)
	at org.eclipse.swt.widgets.Display.syncExec(Unknown Source)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.runTest(E4Testable.java:118)
	at org.eclipse.tycho.surefire.osgibooter.AbstractUITestApplication.runTests(AbstractUITestApplication.java:38)
	at org.eclipse.e4.ui.internal.workbench.swt.E4Testable.lambda$0(E4Testable.java:79)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.mockito.internal.util.concurrent.WeakConcurrentMap.containsKey(WeakConcurrentMap.java:81)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.isMock(MockMethodAdvice.java:166)
	at java.base/java.lang.Object.equals(Object.java:163)
	at org.eclipse.e4.core.internal.di.ObjectDescriptor.hasQualifier(ObjectDescriptor.java:44)
	at org.eclipse.e4.core.internal.di.InjectorImpl.resolveArgs(InjectorImpl.java:573)
	at org.eclipse.e4.core.internal.di.InjectorImpl.invokeUsingClass(InjectorImpl.java:291)
	at org.eclipse.e4.core.internal.di.InjectorImpl.invoke(InjectorImpl.java:233)
	at org.eclipse.e4.core.contexts.ContextInjectionFactory.invoke(ContextInjectionFactory.java:174)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.isEnabled(HandlerServiceHandler.java:68)
	at org.eclipse.core.commands.Command.isEnabled(Command.java:832)
	at org.eclipse.ui.internal.actions.CommandAction.lambda$0(CommandAction.java:91)
	at org.eclipse.ui.internal.actions.CommandAction$$Lambda$384/0x0000000801000d78.commandChanged(Unknown Source)
	at org.eclipse.core.commands.Command$1.run(Command.java:529)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:47)
	at org.eclipse.core.commands.Command.fireCommandChanged(Command.java:522)
	at org.eclipse.core.commands.Command.lambda$0(Command.java:1000)
	at org.eclipse.core.commands.Command$$Lambda$216/0x0000000800edd2b8.handlerChanged(Unknown Source)
	at org.eclipse.core.commands.AbstractHandler.fireHandlerChanged(AbstractHandler.java:77)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.fireHandlerChanged(HandlerServiceHandler.java:189)
	at org.eclipse.core.commands.AbstractHandler.setBaseEnabled(AbstractHandler.java:111)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.isEnabled(HandlerServiceHandler.java:70)
	at org.eclipse.core.commands.Command.isEnabled(Command.java:832)
	at org.eclipse.ui.internal.actions.CommandAction.lambda$0(CommandAction.java:91)
	at org.eclipse.ui.internal.actions.CommandAction$$Lambda$384/0x0000000801000d78.commandChanged(Unknown Source)
	at org.eclipse.core.commands.Command$1.run(Command.java:529)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:47)
	at org.eclipse.core.commands.Command.fireCommandChanged(Command.java:522)
	at org.eclipse.core.commands.Command.lambda$0(Command.java:1000)
	at org.eclipse.core.commands.Command$$Lambda$216/0x0000000800edd2b8.handlerChanged(Unknown Source)
	at org.eclipse.core.commands.AbstractHandler.fireHandlerChanged(AbstractHandler.java:77)
	at org.eclipse.e4.core.commands.internal.HandlerServiceHandler.fireHandlerChanged(HandlerServiceHandler.java:189)
	at org.eclipse.ui.internal.handlers.E4HandlerProxy.handlerChanged(E4HandlerProxy.java:116)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Active Thread: Equinox Container: 882cd8a6-664e-4aea-971d-49c2b2d63bef"
Exception in thread "Worker-4" java.lang.OutOfMemoryError: Java heap space
An error has occurred. See the log file
/home/jenkins/agent/workspace/eclipse.platform.ui_PR-2224/tests/org.eclipse.ui.tests/target/work/data/.metadata/.log.

Also in https://ci.eclipse.org/platform/job/eclipse.platform.ui/view/change-requests/job/PR-2424/11/consoleFull

  WorkbookEditorsHandlerTest.doTearDown:73 » Resource Problems encountered while deleting resources.

Tests run: 1659, Failures: 1, Errors: 5, Skipped: 196

Exception in thread "Active Thread: Equinox Container: 5e5cc6b7-0a59-4770-ba00-8313910b610d" java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-5"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Worker-15"

[merks]

Maybe this is related:

https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/5056

[laeubi]

If I look at this where we exchaust thread pools:

• "Problems encountered while deleting files" due "Thread limit exceeded replacing blocked worker" eclipse.platform#1592

I just wanted to mention that each thread requires a small amount of ram as well....

[iloveeclipse]

I just wanted to mention that each thread requires a small amount of ram as well....

I would expect jenkins runs on a very small VM with very few (2-4?) cores, so thread pool size would be also very small and shouldn't be the root cause here.

[iloveeclipse]

Maybe this is related:
https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/5056

Hmm. I don't see OOM's on JDT tests which are known to be memory sensitive, and OOMs in UI tests seem to happen on / after specific testsuite, so it look like a code regression on our (platform / UI) side.

[laeubi]

I would expect jenkins runs on a very small VM with very few (2-4?) cores, so thread pool size would be also very small and shouldn't be the root cause here.

This is a common misunderstanding with ForkJoinPools... If threads are blocked (e.g. the are waiting for I/O or waiting for other threads to join), then new threads are spwaned untill the configured concurrency level is fulfilled again, see here

The pool attempts to maintain enough active (or available) threads by dynamically adding, suspending, or resuming internal worker threads, even if some tasks are stalled waiting to join others.

The exception given here says "Thread limit exceeded replacing blocked worker" and the limit is by default 32767 (!)

[iloveeclipse]

This is a common misunderstanding with ForkJoinPools

Cool, thanks. So one part of the problem in eclipse-platform/eclipse.platform#1592 is that the used pool is not limited in size (beside that the LocalFile.internalDelete() doesn't do what it supposed to do)?

[laeubi]

It is just a guess, if we see Thread explosion somehwere and memmory problems elsewhere, In general I/O bound tasks don'T play well with fork-join because the whole fork-join was introduced because of CPU bound (!) task usually suffer from context switch penalities. For I/O non blocking I/O is usually most faster if we asumme a shared device (e.g. harddisk / network / ...).

[iloveeclipse]

https://ci.eclipse.org/platform/job/eclipse.platform.ui/job/PR-2433/1/console shows no major memory increase after multiple Thread limit exceeded replacing blocked worker errors.
So something else must be the problem here.