Incoming requests from Zapier are ignored by tracing

[okovpashko]

Hello,

I have a Node.js application that receives webhooks from Zapier. After enabling HTTP tracing, I noticed no traced requests to the webhook endpoints in Sentry UI. When I tried to reproduce a webhook request from my local machine I got its trace in the UI.

Assuming that a clue might be hidden in the Zapier request details I examined an incoming request. I noticed that it includes the following headers: Sentry-Trace, Traceparent, and Tracestate:

Accept */*
Accept-Encoding gzip, deflate
Connection keep-alive
Content-Length 2
Content-Type application/json; charset=utf-8
Sentry-Trace 47a6a99c106b4ee39a237898c24fa0b5-9b8d53f52c06d6d4-0
Traceparent 00-00000000000000004ed6d3ea138a98dc-4ffb60df8bb193a8-01
Tracestate dd=s:1
User-Agent Zapier
X-Datadog-Parent-Id 5763306661388325800
X-Datadog-Sampling-Priority 1
X-Datadog-Trace-Id 5680960982278117596
X-Forwarded-For 52.1.205.184
X-Forwarded-Host a808-xxx-xxx-xxx-xxx.ngrok-free.app
X-Forwarded-Proto https

After enabling the debug mode for Sentry SDK I found that it recognizes the incoming tracing headers and decides to ignore the transaction:

Sentry Logger [debug]: @opentelemetry/instrumentation-http http instrumentation incomingRequest
Sentry Logger [log]: [Tracing] Inheriting remote parent's sampled decision for POST: false
Sentry Logger [log]: [Tracing] Discarding transaction because a negative sampling decision was inherited or tracesSampleRate is set to 0
Sentry Logger [debug]: Recording is off, propagating context in a non-recording span

Could you please suggest if there is a way to make Sentry SDK ignore the incoming headers from Zapier and send the tracing for the webhook requests?

[mydea]

Hey, so the best solution would be to get zapier to not send this header, if it is not wanted 😅

If this is not possible, you could use tracesSampler and ignore the incoming sampling decision for the endpoint that zapier hits - see https://docs.sentry.io/platforms/javascript/guides/node/configuration/sampling/#setting-a-sampling-function. E.g. if you have POST /api-hit-by-zapier, you could do this:

Sentry.init({
  tracesSampler: ({ name, attributes, parentSampled }) => {
    // only continue parent sampling decision if the route is not /api-hit-by-zapier
    if (typeof parentSampled === 'boolean' && attributes?.['http.route'] !== '/api-hit-by-zapier') {
      return parentSampled;
    }

    // regular sample rate you want to have
    return 0.5;
  }
});

You'll need to debug and see what attributes you can actually filter on, what is available there depends on your setup etc., but something like this should work!

[okovpashko]

I'm unsure if I can demand the Zapier team stop sending the trace headers so I'll have to change the configuration on my side.

Thank you for your suggestion about the custom trace sampler. I will try this solution and then will write back about the outcome.

[mydea]

FWIW We are also looking into other ways to handle this better in the future!