Memory leak in recent releases of Sentry (>= 8.6.0)

[jc3m]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/nextjs

SDK Version

8.7.0

Framework Version

No response

Link to Sentry event

No response

SDK Setup

Sentry.init({
  enabled: Configuration.current.sentry.enabled,
  dsn: Configuration.current.sentry.dsn,

  // Adjust this value in production, or use tracesSampler for greater control
  tracesSampleRate: 0.1,

  // Setting this option to true will print useful information to the console while you're setting up Sentry.
  debug: false,

  replaysOnErrorSampleRate: 1.0,

  // This sets the sample rate to be 10%. You may want this to be 100% while
  // in development and sample at a lower rate in production
  replaysSessionSampleRate: 0.1,

  integrations: [
    Sentry.httpClientIntegration(),
    Sentry.replayIntegration({ maskAllText: true, blockAllMedia: true }),
  ],
});

Steps to Reproduce

Next.js application using "@sentry/nextjs": "8.7.0", (here is a full list of dependencies)

Service is deployed via AWS ECS + Fargate

We noticed that our first deploy following an upgrade from 8.3.0 to 8.6.0 started causing our containers to hit their memory limits and crash + restart. We noticed this behavior happening across two separate Next.js applications / containers that were upgraded to 8.6.0 at the same time.

Expected Result

Containers stay under memory limit.

Actual Result

Here is a memory usage graph from one of our containers. Version 8.3.0 does not appear to contain an issue, version > 8.6.0 does, we did not check versions 8.3.0 or 8.4.0

Here are some logs observed at time of crash:

<--- Last few GCs --->
[6:0x6b26810] 13610819 ms: Mark-Compact 251.0 (258.8) -> 250.0 (258.8) MB, 1783.67 / 0.01 ms  (average mu = 0.472, current mu = 0.230) allocation failure; GC in old space requested
[6:0x6b26810] 13613120 ms: Mark-Compact 250.0 (258.8) -> 249.9 (259.1) MB, 1992.86 / 0.00 ms  (average mu = 0.336, current mu = 0.134) allocation failure; GC in old space requested
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xc99970 node::Abort() [next-server (v14.2.2)]
 2: 0xb6ffcb  [next-server (v14.2.2)]
 3: 0xebe9f0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [next-server (v14.2.2)]
 4: 0xebecd7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [next-server (v14.2.2)]
 5: 0x10d0785  [next-server (v14.2.2)]
 6: 0x10e8608 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [next-server (v14.2.2)]
 7: 0x10be721 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [next-server (v14.2.2)]
 8: 0x10bf8b5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [next-server (v14.2.2)]
 9: 0x109bef6 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [next-server (v14.2.2)]
10: 0x108d53c v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawArray(int, v8::internal::AllocationType) [next-server (v14.2.2)]
11: 0x108d923 v8::internal::FactoryBase<v8::internal::Factory>::NewWeakFixedArrayWithMap(v8::internal::Map, int, v8::internal::AllocationType) [next-server (v14.2.2)]
12: 0x10b39c9 v8::internal::Factory::NewTransitionArray(int, int) [next-server (v14.2.2)]
13: 0x13ed56a v8::internal::TransitionsAccessor::Insert(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::SimpleTransitionFlag) [next-server (v14.2.2)]
14: 0x137f750 v8::internal::Map::ConnectTransition(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::SimpleTransitionFlag) [next-server (v14.2.2)]
15: 0x1381672 v8::internal::Map::CopyReplaceDescriptors(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::DescriptorArray>, v8::internal::TransitionFlag, v8::internal::MaybeHandle<v8::internal::Name>, char const*, v8::internal::SimpleTransitionFlag) [next-server (v14.2.2)]
16: 0x1381e56 v8::internal::Map::CopyAddDescriptor(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Descriptor*, v8::internal::TransitionFlag) [next-server (v14.2.2)]
17: 0x1382004 v8::internal::Map::CopyWithField(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::FieldType>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::Representation, v8::internal::TransitionFlag) [next-server (v14.2.2)]
18: 0x1383d0a v8::internal::Map::TransitionToDataProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::PropertyConstness, v8::internal::StoreOrigin) [next-server (v14.2.2)]
19: 0x137327a v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handle<v8::internal::JSReceiver>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::StoreOrigin) [next-server (v14.2.2)]
20: 0x138ecc8 v8::internal::Object::TransitionAndWriteDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::Maybe<v8::internal::ShouldThrow>, v8::internal::StoreOrigin) [next-server (v14.2.2)]
21: 0x1308007 v8::internal::JSObject::AddProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes) [next-server (v14.2.2)]
22: 0x10a94c6 v8::internal::Factory::NewFunctionPrototype(v8::internal::Handle<v8::internal::JSFunction>) [next-server (v14.2.2)]
23: 0xf252c8 v8::internal::Accessors::FunctionPrototypeGetter(v8::Local<v8::Name>, v8::PropertyCallbackInfo<v8::Value> const&) [next-server (v14.2.2)]
24: 0x138ffb5 v8::internal::Object::GetPropertyWithAccessor(v8::internal::LookupIterator*) [next-server (v14.2.2)]
25: 0x11955d8 v8::internal::LoadIC::Load(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>, bool, v8::internal::Handle<v8::internal::Object>) [next-server (v14.2.2)]
26: 0x119af34 v8::internal::Runtime_LoadIC_Miss(int, unsigned long*, v8::internal::Isolate*) [next-server (v14.2.2)]
27: 0x1931ef6  [next-server (v14.2.2)]

<--- Last few GCs --->
--
[14:0x6e58810] 16359129 ms: Scavenge 247.7 (257.6) -> 247.1 (257.8) MB, 2.90 / 0.00 ms  (average mu = 0.206, current mu = 0.068) allocation failure;
[14:0x6e58810] 16359208 ms: Scavenge 247.9 (257.8) -> 247.2 (258.1) MB, 3.32 / 1.43 ms  (average mu = 0.206, current mu = 0.068) allocation failure;
[14:0x6e58810] 16360814 ms: Mark-Compact 248.1 (258.1) -> 246.3 (258.1) MB, 1592.98 / 43.31 ms  (average mu = 0.204, current mu = 0.201) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xc99970 node::Abort() [next-server (v14.2.2)]
2: 0xb6ffcb  [next-server (v14.2.2)]
3: 0xebe9f0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [next-server (v14.2.2)]
4: 0xebecd7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [next-server (v14.2.2)]
5: 0x10d0785  [next-server (v14.2.2)]
6: 0x10d0d14 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [next-server (v14.2.2)]
7: 0x10e7c04 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [next-server (v14.2.2)]
8: 0x10e841c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [next-server (v14.2.2)]
9: 0x10be721 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [next-server (v14.2.2)]
10: 0x10bf8b5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [next-server (v14.2.2)]
11: 0x109ce26 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [next-server (v14.2.2)]
12: 0x14f7c56 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [next-server (v14.2.2)]

[lforst]

Hi, thanks for writing in about this. We are currently taking a look what might cause this!

A question and an ask:

• Would you mind sharing either your sentry.server.config.ts and the contents of instrumentation.ts?
• (Only if it is not too much to ask!) would you mind deploying version 8.4.0 and 8.5.0 to help us narrow down what changes might have introduced the leak.

Also, if you have any custom code around Sentry feel free to share it here!

[AlecRust]

@lforst if it helps, we have the same issue (also using Next.js) and it was caused by upgrading from 8.4.0 to 8.5.0, so I'm fairly confident the issue lies in here.

We are currently pinned to 8.4.0 which doesn't have this issue.

Here is our `instrumentation.ts` if it helps

// This file configures the initialization of Sentry on the server.
// The config you add here will be used whenever the server handles a request.
// https://docs.sentry.io/platforms/javascript/guides/nextjs/

import { checkIfKnownBot, cleanSentryEvent } from '@scripts/clean-sentry-events'
import * as Sentry from '@sentry/nextjs'

/* eslint no-process-env: "off" */
const SENTRY_DSN = process.env.SENTRY_DSN || process.env.NEXT_PUBLIC_SENTRY_DSN

export async function register() {
	if (process.env.NEXT_RUNTIME === 'nodejs') {
		Sentry.init({
			dsn: SENTRY_DSN,
			tracesSampleRate: 0.25,
			environment: process.env.ENVIRONMENT,
			beforeSend: (...args) => {
				const [event, hint] = args

				// Sentry now treats error information as `unknown` which means that the only way to handle this types is really treat them as any and check for existence on runtime
				// https://github.com/getsentry/sentry-javascript/pull/7361

				const originalException = hint?.originalException as any

				const isKnownBot = checkIfKnownBot(event.request?.headers?.['user-agent'])
				if (isKnownBot) {
					return null
				}

				const mainAndMaybeCauseErrors = event.exception?.values?.slice(-2) ?? []

				for (const error of mainAndMaybeCauseErrors) {
					const { type } = error
					const is404 = type === 'Not Found'
					const is403 = type === 'Forbidden'
					const is401 = type === 'Unauthorized'
					const isStripeCardError = originalException?.data?.reason?.raw?.type === 'card_error'
					if (is404 || is401 || is403 || isStripeCardError) {
						return null
					}
				}

				return cleanSentryEvent('server', ...args)
			},
			sendDefaultPii: false,
			ignoreErrors: [
				/**
				 * Error when the user is using a invalid JWT or an JWT that points to a different user.
				 * This is not actionable and is often caused by employee using multiple accounts at the same time.
				 */
				/Not Found/i,
				/Forbidden/i,
				/Unauthorized/i,

				/**
				 * Error when we check an expired or malformed token. This is not actionable.
				 */
				/Received invalid bearer token/i,
			],
			integrations: [
				Sentry.captureConsoleIntegration({
					levels: ['error', 'warn'],
				}),
			],
		})
	}

	if (process.env.NEXT_RUNTIME === 'edge') {
		Sentry.init({
			dsn: SENTRY_DSN,
			tracesSampleRate: 0.25,
			environment: process.env.ENVIRONMENT,
			beforeSend: (...args) => {
				const [event] = args
				const isKnownBot = checkIfKnownBot(event.extra?.userAgent as string)
				if (isKnownBot) {
					return null
				}
				return cleanSentryEvent('edge', ...args)
			},
			sendDefaultPii: false,
			ignoreErrors: [
				/**
				 * Error when the user is using a invalid JWT or an JWT that points to a different user.
				 * This is not actionable and is often caused by employee using multiple accounts at the same time.
				 */
				/Not Found/i,
				/Forbidden/i,
				/Unauthorized/i,

				/**
				 * Error when we check an expired or malformed token. This is not actionable.
				 */
				/Received invalid bearer token/i,

				/**
				 * Validation of expired tokens are not actionable
				 */
				/"exp" claim timestamp check failed/i,
			],
			integrations: [],
		})
	}
}

[PodStuart]

Also seeing this on a bump from 8.3.0 to 8.5.0 last week.

[lforst]

@AlecRust that helps a lot narrowing it down. Thanks! I'll investigate further.