Skip to content

[Python] Enhancements to Queue instrumentation #2951

@antonpirker

Description

@antonpirker

We want to improve the data we collect for task queues.
We will start with enhancing our Celery integration. This should work with redis, and rabbitmq, and as a stretch goal amazon sqs as brokers. (Kafka should follow later.)

How spans are right now:

  • Producer creates <span op="queue.submit.celery"> when task is put into queue. (start_timestamp is when putting into queue is started and end_timestamp is when the item is in the queue.)
  • Consumer creates <transaction op="queue.task.celery"> when task is fetched and processed. (start_timestamp is when the item was received from the queue, end_timestamp is when the processing of the task has finished. )
    • Transaction has a status:
      • ok in case of success
      • aborted if Celery raises Retry, Ignore or Reject celery exceptions
      • internal_error if an unhandled exception was raised during execution of the task

New data we want to add to queue spans:

(for naming see: https://opentelemetry.io/docs/specs/semconv/attributes-registry/messaging/)

  • retry count (can be found in args of existing _wrap_tracer, the attached PR)
  • byte size of message (messaging.message.body.size)
  • byte size of message plus metadata (messaging.message.envelope.size)
  • success/failure status - already there
  • host (server.address) - (can be found in args of existing _wrap_tracer, the attached PR)
  • queue or topic (messaging.destination.name) - not sure where to get this.
  • message id (messaging.message.id) - should also be in args of the comments in the attached PR)
  • consumer id (unique id of the consumer) (messaging.client_id) - dont know yet where to get.
  • what broker is used in celery. (messaging.system)

See this PR for some points where to get the data: #2962

I also added a sample project that uses RabbitMQ with Celery: https://github.com/antonpirker/testing-sentry/tree/main/test-celery-queues-module

Mock of how it will look in the product:

image

### Tasks
- [ ] https://github.com/getsentry/sentry-python/issues/2959

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions