OOM after upgrade from 4.25 to 5.11

[kubijo]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

This is a followup to #25858 with hopes that we can get to the bottom of the memory problems with new Gatsby versions.

While I don't know whether I'll be able to provide minimal repro, I do have a repository that:

• despite myriad other problems, did build fine without much of footprint previously
• in gatsby develop
  • does not build without flags: { DEV_SSR: true }
  • and with it, it gobbles all memory in little time (which amounts, for example, to 40 GB on my current system)
• does not build … no idea here from which end should I approach the debug, so I'm going from the dev side…

Since he requested as much before closing the previous issue, I will mention @pvdz
I'm fully aware that not providing a minimal repro is not an ideal start, but it seems to me that getting that amounts to practically getting the solution… I am open to suggestions & requests for further debug information that would point you to a place of the memory leak... I'd say that even some more interactive debug session might not be out of question (at least timing-wise) if the desire would be there and if Peter is indeed still in Netherlands (Prague here)

Reproduction Link

…

Steps to Reproduce

...

Expected Result

Finishing builds regardless of used mode

Actual Result

build

 ERROR #98123  WEBPACK.BUILD-HTML

Generating SSR bundle failed

Reading from "node:events" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.

File: node:events


 ERROR #98123  WEBPACK.BUILD-HTML

Generating SSR bundle failed

Reading from "node:stream" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.

File: node:stream


 ERROR #98123  WEBPACK.BUILD-HTML

Generating SSR bundle failed

Reading from "node:string_decoder" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.

File: node:string_decoder

not finished Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs - 66.712s
not finished Building HTML renderer - 25.685s

develop without dev_ssr

warn `isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/home/pepa/dev/braiins/academy/node_modules/@babel/types/lib/validators/generated/index.js:2740:35)
    at PluginPass.Program (/home/pepa/dev/braiins/academy/node_modules/babel-plugin-lodash/lib/index.js:102:44)
⠧ Building development bundle
[=========================== ]   5.369 s 62/63 98% Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs

<--- Last few GCs --->

[135252:0x65cc6a0]   102034 ms: Scavenge (reduce) 8176.2 (8215.4) -> 8175.5 (8215.9) MB, 0.7 / 0.0 ms  (average mu = 0.305, current mu = 0.370) allocation failure; 
[135252:0x65cc6a0]   102156 ms: Scavenge (reduce) 8176.3 (8215.9) -> 8175.6 (8215.9) MB, 5.2 / 0.0 ms  (average mu = 0.305, current mu = 0.370) allocation failure; 
[135252:0x65cc6a0]   102163 ms: Scavenge (reduce) 8176.4 (8215.9) -> 8175.7 (8216.2) MB, 4.7 / 0.0 ms  (average mu = 0.305, current mu = 0.370) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xbe3c50 node::Abort() [/home/pepa/.virtualenvs/academy/bin/node]
 2: 0xaf08b4  [/home/pepa/.virtualenvs/academy/bin/node]
 3: 0xdcf7c0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/home/pepa/.virtualenvs/academy/bin/node]
 4: 0xdcfb76 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/home/pepa/.virtualenvs/academy/bin/node]
 5: 0xfce5f5  [/home/pepa/.virtualenvs/academy/bin/node]
 6: 0xfceba6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/home/pepa/.virtualenvs/academy/bin/node]
 7: 0xfe0e96  [/home/pepa/.virtualenvs/academy/bin/node]
 8: 0xfe1ad5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/pepa/.virtualenvs/academy/bin/node]
 9: 0xfbda8f v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/pepa/.virtualenvs/academy/bin/node]
10: 0xfbead7 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/home/pepa/.virtualenvs/academy/bin/node]
11: 0xf9e23a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/home/pepa/.virtualenvs/academy/bin/node]
12: 0x13b1b6f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/home/pepa/.virtualenvs/academy/bin/node]
13: 0x18383f9  [/home/pepa/.virtualenvs/academy/bin/node]

develop with dev_ssr

Killed after some time by system OOM killer.
There is one process that stands out by consuming outrages amounts of memory with command-line attribute reported as .cache/tmp-xxx (by the system-monitor Ubuntu utility)

Environment

System:
    OS: Linux 6.2 Ubuntu 23.04 23.04 (Lunar Lobster)
    CPU: (32) x64 AMD Ryzen 9 7950X 16-Core Processor
    Shell: 5.2.15 - /bin/bash
  Binaries:
    Node: 19.6.0 - /tmp/xfs-2a7e1cd5/node
    Yarn: 3.6.1 - /tmp/xfs-2a7e1cd5/yarn
    npm: 9.4.0 - ~/.virtualenvs/academy/bin/npm
  Languages:
    Python: 3.11.2 - /home/pepa/.virtualenvs/academy/bin/python
  npmPackages:
    gatsby: 5.11.0 => 5.11.0
    gatsby-plugin-algolia: 1.0.3 => 1.0.3
    gatsby-plugin-catch-links: 5.11.0 => 5.11.0
    gatsby-plugin-extract-schema: github:NickyMeuleman/gatsby-plugin-extract-schema#1b6a053ec0a759c94bc6616a68193ad23a9ef6f7 => 0.2.1
    gatsby-plugin-gatsby-cloud: 5.11.0 => 5.11.0
    gatsby-plugin-google-tagmanager: 5.11.0 => 5.11.0
    gatsby-plugin-i18n: 1.0.1 => 1.0.1
    gatsby-plugin-manifest: 5.11.0 => 5.11.0
    gatsby-plugin-mdx: 5.11.0 => 5.11.0
    gatsby-plugin-sass: 6.11.0 => 6.11.0
    gatsby-plugin-sharp: 5.11.0 => 5.11.0
    gatsby-plugin-sitemap: 6.11.0 => 6.11.0
    gatsby-plugin-slug: 2.0.0 => 2.0.0
    gatsby-plugin-typescript: 5.11.0 => 5.11.0
    gatsby-remark-copy-linked-files: 6.11.0 => 6.11.0
    gatsby-remark-images: 7.11.0 => 7.11.0
    gatsby-remark-responsive-iframe: 6.11.1 => 6.11.1
    gatsby-remark-smartypants: 6.11.0 => 6.11.0
    gatsby-remark-unwrap-images: 1.0.2 => 1.0.2
    gatsby-source-filesystem: 5.11.0 => 5.11.0
    gatsby-transformer-yaml: 5.11.0 => 5.11.0

Config Flags

No response

[ThanosDi]

As far as I understand the latest version of Gatsby uses "webpack": "^5.75.0" but webpack fixed the handle of node:module imports on version v5.78.0

so in theory it could just be a matter of Gatgby updating their internal webpack version.

Meanwhile, could you try to add this to your gatsby-node.ts and let us know if it worked?

import webpack from 'webpack';
...
export const onCreateWebpackConfig: GatsbyNode['onCreateWebpackConfig'] =
    async ({ actions }) => {
        actions.setWebpackConfig({
            plugins: [
                new webpack.NormalModuleReplacementPlugin(
                    /node:/,
                    (resource) => {
                        resource.request = resource.request.replace(
                            /^node:/,
                            ''
                        );
                    }
                ),
            ],
            resolve: {
                fallback: {
                    assert: false,
                    buffer: false,
                    console: false,
                    constants: false,
                    crypto: false,
                    domain: false,
                    events: false,
                    http: false,
                    https: false,
                    os: false,
                    path: false,
                    punycode: false,
                    process: false,
                    querystring: false,
                    stream: false,
                    string_decoder: false,
                    sys: false,
                    timers: false,
                    tty: false,
                    url: false,
                    util: false,
                    vm: false,
                    zlib: false,
                    fs: false,
                    tls: false,
                    net: false,
                },
            },
        });
    };