Sentry HTTP 429 errors causing AWS Lambda InvokeErrror (status 502 response) result

[tmilar]

• Review the documentation: https://docs.sentry.io/
• Search for existing issues: https://github.com/getsentry/sentry-javascript/issues
• Use the latest release: https://github.com/getsentry/sentry-javascript/releases
• Provide a link to the affected event from your Sentry account

Package + Version

• @sentry/browser
• @sentry/node 6.17.6
• raven-js
• raven-node (raven for node)
• other:
  • @sentry/serverless 6.17.6
  • @sentry/tracing 6.17.6

Version:

6.17.6

Description

Sometimes, due to large Sentry API usage, we are getting HTTP 429 error responses with Sentry SDK in our Node.js AWS Lambda functions.

However, the problem is that sometimes we end up getting this error caused by Sentry SDK that results in a seemingly unhandleable InvokeError (resulting in AWS lambda HTTP 502 response) which is problematic for us because our actual business logic is working just fine.

2022-02-15T18:50:47.224Z	67dc62db-a670-4e84-ad4a-679145ecd1e1	ERROR	Invoke Error 	
{
    "errorType": "SentryError",
    "errorMessage": "HTTP Error (429)",
    "name": "SentryError",
    "stack": [
        "SentryError: HTTP Error (429)",
        "    at new SentryError (/var/task/node_modules/@sentry/utils/dist/error.js:9:28)",
        "    at ClientRequest.<anonymous> (/var/task/node_modules/@sentry/node/dist/transports/base/index.js:212:44)",
        "    at Object.onceWrapper (events.js:520:26)",
        "    at ClientRequest.emit (events.js:400:28)",
        "    at ClientRequest.emit (domain.js:475:12)",
        "    at HTTPParser.parserOnIncomingClient (_http_client.js:647:27)",
        "    at HTTPParser.parserOnHeadersComplete (_http_common.js:127:17)",
        "    at TLSSocket.socketOnData (_http_client.js:515:22)",
        "    at TLSSocket.emit (events.js:400:28)",
        "    at TLSSocket.emit (domain.js:475:12)"
    ]
}

Our expectation is that internal Sentry errors should not cause an outage in our own APIs.

We didn't found any way to handle/catch this error, because it looks like it's failing outside of our Sentry calls such as Sentry.init(), Sentry.wrapHandler() etc.

The 429 error response we are mostly certain that is due to an exceess in our Transactions quota.

When decreasing traceSampleRate config in Sentry.init() from 0.2 to 0, we stopped having this error.

We were seeing this issue was happening consistently in random cases, seemingly the same as the value we had for traceSampleRate. We've also tried setting it to 1 to confirm, and we could confirm a 100% error rate.

So currently, the workaround we are using is disabling this entire feature by setting it to 0, and no more unhandled Sentry 429 HTTP errors were thrown. Still, it doesn't seem to be correct solution having to disable entire features to have an external dependency not breaking my app.

For completeness, this is our current config:

    Sentry.init({
      debug: SLS_STAGE === 'dev',
      dsn: sentryKey,
      tracesSampleRate: 0,

      environment: SLS_STAGE,
      release: `${SLS_SERVICE_NAME}:${SLS_APIG_DEPLOYMENT_ID}`,
    })

[AbhiPrasad]

This is the logic that we are hitting.

sentry-javascript/packages/node/src/transports/base/index.ts

Lines 248 to 252 in 1bf9883

	let rejectionMessage = `HTTP Error (${statusCode})`;
	if (res.headers && res.headers['x-sentry-error']) {
	rejectionMessage += `: ${res.headers['x-sentry-error']}`;
	}
	reject(new SentryError(rejectionMessage));

Sentry though doesn't capture internal SentryErrors though. Is AWS Lambda monitoring errors that bubble up to certain handlers and setting a response based on that?

Perhaps there is a way we can edit https://github.com/getsentry/sentry-javascript/blob/master/packages/serverless/src/awslambda.ts to address that. Maybe we could monkey patch whatever aws lambda is listening for to filter out SentryError.

We could also try and not throw errors for 429s since they are pretty common.

What do you think?

[collierrgbsitisfise]

@AbhiPrasad we was facing the same issue. Is is legit way to add additional parameter on Sentry.init or on Sentry.captureError, which will explicitly define if internal(sentry) error should be ignored or not ?

[tmilar]

@AbhiPrasad
Yes, it looks like AWS lambda default behavior is to respond with InvokeError on any unhandled error.

I think that unhandleable errors should not ever happen from an external library.

On the current Senttry implementation. I'm not seeing a way the client could catch this error since it's happening asynchronously, a correct approach in this case would be at least having them absorbed and just logged. Maybe, log it always, or only in case of debug: true, or some other logging flag would be set.

[ichina]

@tmilar hopefully will be fixed with #4620

[tmilar]

Yes @ichina , that probably would work fine. Thanks!