Filter OpenTelemetry traces caused by Sentry

[hkrutzer]

Problem Statement

I am sending OpenTelemetry to a service X that isn't Sentry. I want to start sending it to Sentry as well, but because Sentry only supports sending it via the SDK, I have to add the Sentry spanprocessor. However, this causes additional traces from Sentry itself to show up in X as well (from instrumentation-net and instrumentation-http).

These are unrelated to my application and I don't want to store them. Otel Collector doesn't allow discarding specific spans and their children as far as I can tell, so it would be great if I could filter them from the SDK configuration.

Solution Brainstorm

In https://github.com/getsentry/sentry-javascript/blob/develop/packages/opentelemetry-node/src/spanprocessor.ts#L93-L100 there is some code that prevents HTTP requests to Sentry from being sent as telemetry to Sentry. Perhaps this can be used to prevent these requests from going to other OpenTelemetry services as well.

[mydea]

Problem Statement

I am sending OpenTelemetry to a service X that isn't Sentry. I want to start sending it to Sentry as well, but because Sentry only supports sending it via the SDK, I have to add the Sentry spanprocessor. However, this causes additional traces from Sentry itself to show up in X as well (from instrumentation-net and instrumentation-http).

These are unrelated to my application and I don't want to store them. Otel Collector doesn't allow discarding specific spans and their children as far as I can tell, so it would be great if I could filter them from the SDK configuration.

Solution Brainstorm

In https://github.com/getsentry/sentry-javascript/blob/develop/packages/opentelemetry-node/src/spanprocessor.ts#L93-L100 there is some code that prevents HTTP requests to Sentry from being sent as telemetry to Sentry. Perhaps this can be used to prevent these requests from going to other OpenTelemetry services as well.

Hey,

could you paste your Sentry.init config? Did you set instrumenter: 'otel' as described in the docs here: https://docs.sentry.io/platforms/node/performance/instrumentation/opentelemetry/? With this (required) setting, Sentry should not generate any traces itself.

[hkrutzer]

Sure, I have a single file where the Otel SDK and Sentry are configured:

const process = require('process');
const opentelemetry = require('@opentelemetry/sdk-node');
const { trace } = require('@opentelemetry/api');
const {
  getNodeAutoInstrumentations,
} = require('@opentelemetry/auto-instrumentations-node');
const {
  BasicTracerProvider,
  SimpleSpanProcessor,
} = require('@opentelemetry/sdk-trace-base');
const {
  OTLPTraceExporter,
} = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const {
  SemanticResourceAttributes,
} = require('@opentelemetry/semantic-conventions');
const Sentry = require('@sentry/node');
const {
  SentrySpanProcessor,
  SentryPropagator,
} = require('@sentry/opentelemetry-node');

const OTLP_ENDPOINT = process.env.OTLP_ENDPOINT || 'http://localhost:4318';
const traceExporter = new OTLPTraceExporter({
  url: `${OTLP_ENDPOINT}/v1/traces`,
});

if (process.env.SENTRY_DSN) {
  Sentry.init({
    dsn: process.env.SENTRY_DSN,
    tracesSampleRate: 1.0,
    instrumenter: 'otel',
  });
  Sentry.configureScope((scope) => {
    scope.setTag('environment', process.env.ENVIRONMENT || 'dev');
  });
}

const provider = new BasicTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'servicename',
    env: process.env.ENVIRONMENT || 'dev',
  }),
});
provider.addSpanProcessor(new SentrySpanProcessor());
provider.addSpanProcessor(new SimpleSpanProcessor(traceExporter));
provider.register();
trace.setGlobalTracerProvider(provider);

const sdk = new opentelemetry.NodeSDK({
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': {
        enabled: false,
      },
    }),
  ],

  textMapPropagator: new SentryPropagator(),
});

function init() {
  sdk.start();
}

init();

[mydea]

Hmm, seems correct at a glance. And what kind of incorrect spans are you getting? Something auto instrumented?

The only thing I can see is that the OTEL http instrumentation will also create spans for the outgoing sentry requests. These you can filter like this:

const sdk = new opentelemetry.NodeSDK({
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-http': {
         ignoreOutgoingRequestHook: request => {
            const host = request.host || request.hostname;
             const dsn = getCurrentHub().getClient()?.getDsn();
             return dsn && host ? host.includes(dsn.host) : false;
          },
      },
    }),
  ],
});

[hkrutzer]

Indeed those are the ones I am getting, as well as the same from instrumentation-net, which doesn't have a filter function.