fix(kafkajs): add APM tracing support for batch consumer #6937
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The batch consumer plugin was only handling Data Streams Monitoring (DSM) but not creating APM trace spans. This adds the missing bindStart() method to create spans for eachBatch callbacks, similar to how the regular consumer plugin creates spans for eachMessage callbacks. Changes: - Add bindStart() method to create APM trace spans for batch consumption - Extract parent context from first message headers for distributed tracing - Add span tags: kafka.topic, kafka.partition, kafka.batch_size - Add kafka.first_offset and kafka.last_offset tags for batch boundaries - Set span resource to topic name and type to 'worker' - Integrate DSM checkpoints with APM spans (pass span to setCheckpoint) - Add comprehensive test coverage for batch consumer APM tracing This fix ensures that batch consumer operations are properly traced in APM, providing visibility into batch consumption performance and enabling distributed tracing across Kafka producers and consumers. Fixes missing APM spans for @confluentinc/kafka-javascript and kafkajs batch consumers when using eachBatch callbacks.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The batch consumer plugin was only handling Data Streams Monitoring (DSM) but not creating APM trace spans. This adds the missing bindStart() method to create spans for eachBatch callbacks, similar to how the regular consumer plugin creates spans for eachMessage callbacks.
Changes:
This fix ensures that batch consumer operations are properly traced in APM, providing visibility into batch consumption performance and enabling distributed tracing across Kafka producers and consumers.
Fixes missing APM spans for @confluentinc/kafka-javascript and kafkajs batch consumers when using eachBatch callbacks.
What does this PR do?
Motivation
Plugin Checklist
Additional Notes