On un-sleeping laptop, element desktop sometimes goes entirely white (because EW has OOMed during the night)

[ara4n]

Steps to reproduce

1. Unsleep mac laptop while running nightly
2. Observe that the app is entirely blank (and inspector won't open)

The app quite cleanly via File: Quit however.

Will send logs at next launch.

Outcome

What did you expect?

No white

What happened instead?

All the white

Operating system

macOS 12.3 on M1

Application version

Nightly

How did you install the app?

nightly

Homeserver

matrix.org

Will you send logs?

Yes

[ara4n]

I am currently getting this almost every morning.

[ara4n]

Yup, i've had this every single time i've unslept my laptop over the last week.

[ara4n]

See also #672 and #824 and electron/electron#2357.

I am getting this literally every day i open my M1 mac. @novocaine says here that:

A white screen means Electron's renderer process has crashed. Element code can't crash that process as it runs in a container - there's some problem with either Electron or your system (drivers, run out of resources etc).

Just for some additional diagnostics, the pstree I have running while it's stuck in this state are:

 |-+= 33570 matthew /Applications/Element Nightly.app/Contents/MacOS/Element Nightly
 | |--- 33571 matthew /Applications/Element Nightly.app/Contents/Frameworks/Element Nightly Helper (GPU).app/Contents/MacOS/Element Nightly Helper (GPU) --type=gpu-process --user-data-dir=/Users/matthew/Library/Application Support/Element Nightly --gpu-preferences=UAAAAAAAAAAgAAAAAAAAAAAAAAAAAAAAAABgAAAAAAAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJgEAAAAAAAAmAQAAAAAAACIAQAAMAAAAIABAAAAAAAAiAEAAAAAAACQAQAAAAAAAJgBAAAAAAAAoAEAAAAAAACoAQAAAAAAALABAAAAAAAAuAEAAAAAAADAAQAAAAAAAMgBAAAAAAAA0AEAAAAAAADYAQAAAAAAAOABAAAAAAAA6AEAAAAAAADwAQAAAAAAAPgBAAAAAAAAAAIAAAAAAAAIAgAAAAAAABACAAAAAAAAGAIAAAAAAAAgAgAAAAAAACgCAAAAAAAAMAIAAAAAAAA4AgAAAAAAAEACAAAAAAAASAIAAAAAAABQAgAAAAAAAFgCAAAAAAAAYAIAAAAAAABoAgAAAAAAAHACAAAAAAAAeAIAAAAAAACAAgAAAAAAAIgCAAAAAAAAkAIAAAAAAACYAgAAAAAAAKACAAAAAAAAqAIAAAAAAACwAgAAAAAAALgCAAAAAAAAwAIAAAAAAADIAgAAAAAAANACAAAAAAAA2AIAAAAAAADgAgAAAAAAAOgCAAAAAAAA8AIAAAAAAAD4AgAAAAAAABAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAHAAAAEAAAAAAAAAAAAAAACAAAABAAAAAAAAAAAAAAAAkAAAAQAAAAAAAAAAAAAAALAAAAEAAAAAAAAAAAAAAADAAAABAAAAAAAAAAAAAAAA4AAAAQAAAAAAAAAAAAAAAPAAAAEAAAAAAAAAABAAAAAAAAABAAAAAAAAAAAQAAAAcAAAAQAAAAAAAAAAEAAAAIAAAAEAAAAAAAAAABAAAACQAAABAAAAAAAAAAAQAAAAsAAAAQAAAAAAAAAAEAAAAMAAAAEAAAAAAAAAABAAAADgAAABAAAAAAAAAAAQAAAA8AAAAQAAAAAAAAAAQAAAAAAAAAEAAAAAAAAAAEAAAABwAAABAAAAAAAAAABAAAAAgAAAAQAAAAAAAAAAQAAAAJAAAAEAAAAAAAAAAEAAAACwAAABAAAAAAAAAABAAAAAwAAAAQAAAAAAAAAAQAAAAOAAAAEAAAAAAAAAAEAAAADwAAABAAAAAAAAAABwAAAAAAAAAQAAAAAAAAAAcAAAAHAAAAEAAAAAAAAAAHAAAACAAAABAAAAAAAAAABwAAAAkAAAAQAAAAAAAAAAcAAAALAAAAEAAAAAAAAAAHAAAADAAAABAAAAAAAAAABwAAAA4AAAAQAAAAAAAAAAcAAAAPAAAAEAAAAAAAAAAIAAAAAAAAABAAAAAAAAAACAAAAAcAAAAQAAAAAAAAAAgAAAAIAAAAEAAAAAAAAAAIAAAACQAAABAAAAAAAAAACAAAAAsAAAAQAAAAAAAAAAgAAAAMAAAAEAAAAAAAAAAIAAAADgAAABAAAAAAAAAACAAAAA8AAAAQAAAAAAAAAAoAAAAAAAAAEAAAAAAAAAAKAAAABwAAABAAAAAAAAAACgAAAAgAAAAQAAAAAAAAAAoAAAAJAAAAEAAAAAAAAAAKAAAACwAAABAAAAAAAAAACgAAAAwAAAAQAAAAAAAAAAoAAAAOAAAAEAAAAAAAAAAKAAAADwAAAAgAAAAAAAAACAAAAAAAAAA= --shared-files --field-trial-handle=1718379636,9501207313218371973,7896253060103614826,131072 --enable-features=WebRTCPipeWireCapturer --disable-features=HardwareMediaKeyHandling,MediaSessionService,PlzServiceWorker,SpareRendererForSitePerProcess --seatbelt-client=40
 | \--- 33572 matthew /Applications/Element Nightly.app/Contents/Frameworks/Element Nightly Helper.app/Contents/MacOS/Element Nightly Helper --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-GB --service-sandbox-type=network --user-data-dir=/Users/matthew/Library/Application Support/Element Nightly --standard-schemes=vector --enable-sandbox --secure-schemes=vector --bypasscsp-schemes --cors-schemes --fetch-schemes=vector --service-worker-schemes --streaming-schemes --shared-files --field-trial-handle=1718379636,9501207313218371973,7896253060103614826,131072 --enable-features=WebRTCPipeWireCapturer --disable-features=HardwareMediaKeyHandling,MediaSessionService,PlzServiceWorker,SpareRendererForSitePerProcess --seatbelt-client=41

although fwiw Instruments (having sampled via the Activity Monitor instrument) also claims there's MTLCompilerService and VTDecoderXPCService and VTEncoderXPCServicer with a "responsible process" of 33570 (the main Element Nightly).

Meanwhile, taking a thread dump of the three main processes via the Activity Monitor app (not Instruments) gives:

nightly-sample.txt
nightly-helper-sample.txt
nightly-helper-gpu-sample.txt

Ah - I just found the crash log for the renderer process, which apparently exited with SIGTRAP (EXC_BREAKPOINT?!) at 02:43:08, when i was asleep:

nightly-helper-renderer-crashdump.txt

For the sake of completeness, here are the last few other crashdumps from the same in the middle of the night:

Element Nightly Helper (Renderer)-2022-05-04-043838.ips.txt
Element Nightly Helper (Renderer)-2022-05-07-025828.ips.txt

Looking at the JSON stackstraces in these thread dumps:

crash.json.txt

...it looks like thread ID 1248168 is the one which crashed, due to v8 running out of memory (while trying to GC):

  {
    "triggered": true,
    "id": 1248168,
    "name": "ThreadPoolForegroundWorker",
    "threadState": {
      "x": [
        {
          "value": 4732437384
        },
        {
          "value": 105553245181312
        },
        {
          "value": 1024
        },
        {
          "value": 6272652880
        },
        {
          "value": 6272652032
        },
        {
          "value": 6272651744
        },
        {
          "value": 729647118309407400
        },
        {
          "value": 1280
        },
        {
          "value": 0
        },
        {
          "value": 0
        },
        {
          "value": 0
        },
        {
          "value": 2
        },
        {
          "value": 2
        },
        {
          "value": 0
        },
        {
          "value": 0
        },
        {
          "value": 0
        },
        {
          "value": 7241649484,
          "symbolLocation": 0,
          "symbol": "pthread_mutex_unlock"
        },
        {
          "value": 6272692224
        },
        {
          "value": 0
        },
        {
          "value": 6272653200
        },
        {
          "value": 6272653216
        },
        {
          "value": 6272653224
        },
        {
          "value": 6272651880
        },
        {
          "value": 6272688863
        },
        {
          "value": 4731625472,
          "symbolLocation": 15664,
          "symbol": "v8_inspector::protocol::Debugger::API::Paused::ReasonEnum::XHR"
        },
        {
          "value": 56
        },
        {
          "value": 8683873544,
          "symbolLocation": 0,
          "symbol": "__stderrp"
        },
        {
          "value": 534836150272
        },
        {
          "value": 134227373
        }
      ],
      "flavor": "ARM_THREAD_STATE64",
      "lr": {
        "value": 4651545376
      },
      "cpsr": {
        "value": 1610616832
      },
      "fp": {
        "value": 6272653184
      },
      "sp": {
        "value": 6272651856
      },
      "esr": {
        "value": 4060086272,
        "description": "(Breakpoint) brk 0"
      },
      "pc": {
        "value": 4651545904,
        "matchesCrashFrame": 1
      },
      "far": {
        "value": 6272653624
      }
    },
    "frames": [
      {
        "imageOffset": 42005808,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 8767684,
        "imageIndex": 3
      },
      {
        "imageOffset": 42005280,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 8767156,
        "imageIndex": 3
      },
      {
        "imageOffset": 60872656,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 27634532,
        "imageIndex": 3
      },
      {
        "imageOffset": 13763320,
        "symbol": "v8::internal::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*)",
        "symbolLocation": 724,
        "imageIndex": 3
      },
      {
        "imageOffset": 13763112,
        "symbol": "v8::internal::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*)",
        "symbolLocation": 516,
        "imageIndex": 3
      },
      {
        "imageOffset": 15541168,
        "symbol": "v8::internal::Heap::StartIncrementalMarking(int, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)",
        "symbolLocation": 1108,
        "imageIndex": 3
      },
      {
        "imageOffset": 16367628,
        "symbol": "v8::internal::GlobalSafepointScope::~GlobalSafepointScope()",
        "symbolLocation": 241132,
        "imageIndex": 3
      },
      {
        "imageOffset": 16183292,
        "symbol": "v8::internal::GlobalSafepointScope::~GlobalSafepointScope()",
        "symbolLocation": 56796,
        "imageIndex": 3
      },
      {
        "imageOffset": 16128432,
        "symbol": "v8::internal::GlobalSafepointScope::~GlobalSafepointScope()",
        "symbolLocation": 1936,
        "imageIndex": 3
      },
      {
        "imageOffset": 16127796,
        "symbol": "v8::internal::GlobalSafepointScope::~GlobalSafepointScope()",
        "symbolLocation": 1300,
        "imageIndex": 3
      },
      {
        "imageOffset": 60775440,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 27537316,
        "imageIndex": 3
      },
      {
        "imageOffset": 60770400,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 27532276,
        "imageIndex": 3
      },
      {
        "imageOffset": 42241136,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 9003012,
        "imageIndex": 3
      },
      {
        "imageOffset": 42376056,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 9137932,
        "imageIndex": 3
      },
      {
        "imageOffset": 42373568,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 9135444,
        "imageIndex": 3
      },
      {
        "imageOffset": 42559652,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 9321528,
        "imageIndex": 3
      },
      {
        "imageOffset": 42373028,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 9134904,
        "imageIndex": 3
      },
      {
        "imageOffset": 42432288,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 9194164,
        "imageIndex": 3
      },
      {
        "imageOffset": 42431332,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 9193208,
        "imageIndex": 3
      },
      {
        "imageOffset": 42560896,
        "symbol": "node::GetEnvironmentIsolateData(node::Environment*)",
        "symbolLocation": 9322772,
        "imageIndex": 3
      },
      {
        "imageOffset": 29292,
        "symbol": "_pthread_start",
        "symbolLocation": 148,
        "imageIndex": 6
      },
      {
        "imageOffset": 8332,
        "symbol": "thread_start",
        "symbolLocation": 8,
        "imageIndex": 6
      }
    ]
  },

So: what could be causing an explosion of v8 memory utilisation on the renderer process, and how do I stop it?

[ara4n]

So I just got this again, except this time the inspector was open. This didn't help much, given the inspector disconnected from the app when the renderer crashed, however I think I have an accurate memory heap size estimate from the point of the crash (701MB)...

...although this probably don't reflect the sudden memory spike which killed it.

The crash happened at 05:27:15 GMT+1 according to the crash log:

Element Nightly Helper (Renderer)-2022-05-10-052716.ips.txt

The last thing on the console was:

05:11:43.118 rageshake.ts:72 got device keys for @ajbura:matrix.org: {CEWMASBLTX: {…}, IBHOESCDUX: {…}, IXANQRLYTC: {…}, KRLRINCZCW: {…}, OBVENZDKQL: {…}, …}
05:11:43.118 rageshake.ts:72 got cross-signing keys for @ajbura:matrix.org: {master: {…}, self_signing: {…}, user_signing: undefined}
05:11:43.121 rageshake.ts:72 Completed key download for @ajbura:matrix.org
05:11:43.121 rageshake.ts:72 Device list for @ajbura:matrix.org now up to date
05:11:43.705 ://vector/webapp/872e090734592b905ed9.worker.js:885 Persisting sync data up to m2928537925~3.2928537933~36.2928537933~2.2928537933_757284974_46494294_1350728213_1364023735_3238494_483925928_3618151627_212414
05:11:44.311 rageshake.ts:72 Saving device tracking data m2928537925~3.2928537933~36.2928537933~2.2928537933_757284974_46494294_1350728213_1364023735_3238494_483925928_3618151627_212414

...so significantly before the crash.

[ara4n]

Options for debugging this further:

• Get a coredump and hope that lldb can walk its v8 stack, or failing that manually look in the data of the native stack to see what the exploding thread was up to. Trying this tonight via https://developer.apple.com/forums/thread/694233
• Monitor the native RSZ and SIGSTOP when it spikes so you can then hopefully attach the chrome inspector and breakpoint it before it dies
• Figure out how to monitor the v8 heap from within electron and call debugger before it dies.
• Disable SIP and try to run it under lldb?
• Build a debug electron, and run that under lldb?

---

Also discussed at https://matrix.to/#/!zKcwXhRfcrDOBEYsgu:matrix.org/$fHG1BhGvNW4-Z-LvRjVowzVHrelDewTWzJs7J7_0j6A?via=matrix.org&via=element.io&via=vector.modular.im

[MadLittleMods]

@ara4n Please document any debugging steps to get any of those options working. I'm extremely interested in how to get information for scenarios like these as well.

[ara4n]

So it crashed again last night (at 03:22:35), but it didn't dump a core (despite following @justkwin's instructions at https://developer.apple.com/forums/thread/694233). However, did get another datapoint simply from having run Electron from the commandline, which is that its dying words were:

<--- Last few GCs --->

[48339:0x158050000] 15205366 ms: Scavenge 2986.0 (3102.8) -> 2984.6 (3104.1) MB, 6.5 / 0.0 ms  (average mu = 0.982, current mu = 0.984) allocation failure 
[48339:0x158050000] 15205390 ms: Scavenge 2988.0 (3104.1) -> 2988.1 (3104.1) MB, 16.5 / 0.0 ms  (average mu = 0.982, current mu = 0.984) allocation failure 
[48339:0x158050000] 15205404 ms: Scavenge 2988.3 (3104.1) -> 2988.3 (3104.1) MB, 14.5 / 0.0 ms  (average mu = 0.982, current mu = 0.984) allocation failure 


<--- JS stacktrace --->

[48339:0511/032235.122008:FATAL:v8_initializer.cc(820)] 
Error sending from webFrameMain:  Error: Render frame was disposed before WebFrameMain could be accessed
    at EventEmitter._send (<anonymous>)
    at EventEmitter.n.send (node:electron/js2c/browser_init:165:417)
    at EventEmitter.b.send (node:electron/js2c/browser_init:161:2494)
    at IpcMainImpl.<anonymous> (/Applications/Element Nightly.app/Contents/Resources/app.asar/lib/electron-main.js:803:32)
    at Generator.next (<anonymous>)
    at fulfilled (/Applications/Element Nightly.app/Contents/Resources/app.asar/lib/electron-main.js:42:58)
Error sending from webFrameMain:  Error: Render frame was disposed before WebFrameMain could be accessed
    at EventEmitter.n.send (node:electron/js2c/browser_init:165:417)
    at EventEmitter.b.send (node:electron/js2c/browser_init:161:2494)
    at ipcChannelSendUpdateStatus (/Applications/Element Nightly.app/Contents/Resources/app.asar/lib/updater.js:90:35)
    at AutoUpdater.<anonymous> (/Applications/Element Nightly.app/Contents/Resources/app.asar/lib/updater.js:95:5)
    at AutoUpdater.emit (node:events:390:28)

In other words, v8 heap spiked up to 3.1GB before it died (up from the ~600MB it was when it went to sleep).

I guess i try to find out why it didn't dump a core, or failing that try to run it under lldb, or failing that do the SIGSTOP hack to try to catch it in the act.

[ara4n]

So i can get naive core dumps okay following Apple's instructions:

matthew@shadowfax ~ % cat crash.c          
#include <stdio.h>

int main() {
	char * c = 0;
	printf("%c", *c);
	return(0);
}
matthew@shadowfax ~ % cc -o crash crash.c
matthew@shadowfax ~ % ulimit -c unlimited
matthew@shadowfax ~ % /usr/libexec/PlistBuddy -c "Add :com.apple.security.get-task-allow bool true" tmp.entitlements
File Doesn't Exist, Will Create: tmp.entitlements
matthew@shadowfax ~ % codesign -s - -f --entitlements tmp.entitlements crash
crash: replacing existing signature
matthew@shadowfax ~ % time ./crash
zsh: segmentation fault (core dumped)  ./crash
./crash  0.00s user 0.50s system 61% cpu 0.813 total
matthew@shadowfax ~ % ls -al /cores/core.55325 
-r--------  1 matthew  wheel  3023405056 11 May 09:31 /cores/core.55325
matthew@shadowfax ~ % lldb -c /cores/core.55325 
(lldb) target create --core "/cores/core.55325"
Core file '/cores/core.55325' (arm64) was loaded.
(lldb) bt
* thread vector-im/element-web#1, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x0)
  * frame #0: 0x0000000100377f7c crash`main + 32
    frame vector-im/element-web#1: 0x0000000100695088 dyld`start + 516

[ara4n]

So it looks like the reason the core dumps failed for the renderer process is that i failed to set the entitlements correctly:

matthew@shadowfax MacOS % codesign -s - -f --entitlements tmp.entitlements Element\ Nightly\ Helper\ \(Renderer\) 
Element Nightly Helper (Renderer): replacing existing signature
Element Nightly Helper (Renderer): code object is not signed at all
In subcomponent: /Applications/Element Nightly.app/Contents/Frameworks/Element Nightly Helper (Renderer).app/Contents/MacOS/tmp.entitlements

and "code object is not signed at all" is a cryptic error implying that it couldn't re-sign it for some reason. The binary absolutely is signed though; codesign -dvvvvvvvv gives:

Signature size=9078
Authority=Developer ID Application: Vector Creations Limited (7J4U792NQT)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=11 May 2022 at 08:05:12

Perhaps it needs to be re-signed with a real identity rather than an adhoc identity ("-"). Eitherway, after running codesign -s, the output of codesign -dvvvvvvvvvvvvv Element\ Nightly\ Helper\ \(Renderer\) is the same as before, and the entitlements have not been updated to include the required get-task-allow entitlement:

matthew@shadowfax MacOS % codesign --display --entitlements - Element\ Nightly\ Helper\ \(Renderer\)                    
Executable=/Applications/Element Nightly.app/Contents/Frameworks/Element Nightly Helper (Renderer).app/Contents/MacOS/Element Nightly Helper (Renderer)
[Dict]
	[Key] com.apple.security.cs.allow-jit
	[Value]
		[Bool] true
	[Key] com.apple.security.cs.disable-library-validation
	[Value]
		[Bool] true
	[Key] com.apple.security.cs.allow-unsigned-executable-memory
	[Value]
		[Bool] true

So, either i fight against the signing system or i disable SIP...

[ara4n]

Having poked further at codesign it turns out it barfed because the tmp.entitlements file was in the cwd - the "code object is not signed at all" was referring to the temporary entitlements file. So, on creating a new entitlements file with the additional get-task-allow entitlement:

% for i in cs.allow-jit cs.disable-library-validation cs.allow-unsigned-executable-memory get-task-allow; do /usr/libexec/PlistBuddy -c "Add :com.apple.security.$i bool true" ~/tmp.entitlements; done
File Doesn't Exist, Will Create: /Users/matthew/tmp.entitlements
% cat ~/tmp.entitlements 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>com.apple.security.cs.allow-jit</key>
	<true/>
	<key>com.apple.security.cs.allow-unsigned-executable-memory</key>
	<true/>
	<key>com.apple.security.cs.disable-library-validation</key>
	<true/>
	<key>com.apple.security.get-task-allow</key>
	<true/>
</dict>
</plist>

...and signing it with a real identity (just because i have one easily to hand, thanks to XCode, and because I don't want to get bitten by weird & wonderful adhoc codesigning quirks):

% codesign -s 9A3ZBW6V78 -f --entitlements ~/tmp.entitlements Element\ Nightly\ Helper\ \(Renderer\)                                                                                                            
Element Nightly Helper (Renderer): replacing existing signature
% codesign --display --entitlements - Element\ Nightly\ Helper\ \(Renderer\)
Executable=/Applications/Element Nightly.app/Contents/Frameworks/Element Nightly Helper (Renderer).app/Contents/MacOS/Element Nightly Helper (Renderer)
[Dict]
	[Key] com.apple.security.cs.allow-jit
	[Value]
		[Bool] true
	[Key] com.apple.security.cs.allow-unsigned-executable-memory
	[Value]
		[Bool] true
	[Key] com.apple.security.cs.disable-library-validation
	[Value]
		[Bool] true
	[Key] com.apple.security.get-task-allow
	[Value]
		[Bool] true
% codesign -dvvvvvvvvvvvvvv Element\ Nightly\ Helper\ \(Renderer\) 2>&1 | egrep 'Sign|Auth'
Signature size=4877
Authority=Apple Development: Matthew Hodgson (9A3ZBW6V78)
Authority=Apple Worldwide Developer Relations Certification Authority
Authority=Apple Root CA
Signed Time=11 May 2022 at 10:19:24

...looks much more plausible. So, setting it running again with ulimit -c unlimited to see what happens. It's still a bit risky, given digging around inside the native core could be a bit painful without a debug build of electron, but it might be good enough to spot what v8 is doing. Failing that, it'll be time for a debug electron build as per https://www.nylas.com/blog/low-level-electron-debugging/.

[ara4n]

(of course, having done this, it didn't crash overnight; v8 heap is 700MB in the morning)

[ara4n]

good news - it crashed overnight (2022-05-13 02:48:28.7125 +0100). bad news: it still didn't dump core. Perhaps the rlimits got reset on the child process, or perhaps there's something daft like Termination Reason: Namespace SIGNAL, Code 5 Trace/BPT trap: 5 doesn't actually trigger a core dump.