RFC: OpenTelemetry Integration for Powertools for AWS Lambda - Tracer only #90
Replies: 4 comments 12 replies
-
Great initiative
Additionally I do se a problem with setting the account id as resource attribute because is not available at cold start (init) time. |
Beta Was this translation helpful? Give feedback.
-
Another thing that comes to mind, where the powertools can make a great impact, is the batch processors. In OpenTelemetry there is the concept of span links, which is intended to be used in messaging scenarios.
The problem is that in messaging scenarios you don't want to have all messaging processors in one giant trace. To mitigate that it can be split into multiple ones and link them. I did a POC implementation here |
Beta Was this translation helpful? Give feedback.
-
For the parser utility it should also extract and set the relevant semantic convention attributes. |
Beta Was this translation helpful? Give feedback.
-
Just one thought: The integration should have an easy way to fallback to other tracing mechanisms (i.e. X-Ray, Logging) in case the OTEL SDK or the OTEL Layer is not available. Also, what are the strategies you are considering handling tracing in local environments? There should be a mechanism to fallback to a local configuration in case the user is testing the function and they use a remote OTEL endpoint. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Note
DISCLAIMER
This RFC is open for discussion and we're actively seeking community feedback to shape OpenTelemetry support in Powertools for AWS Lambda.
At this stage we have not committed to implementing it and our plans will vary based on your feedback.
The feature set described here is not final and represents our current preferred approach. We encourage all interested parties to provide feedback both in this RFC or via email at [email protected]
We'll use Python in all the code samples throughout the RFC, but the general ideas and design choices apply to all Powertools for AWS languages. Class and method names are subject to change.
RFC: OpenTelemetry Integration for Powertools for AWS Lambda - Tracer only
Summary
The OpenTelemetry integration for Powertools for AWS Lambda provides a standardized, vendor-agnostic approach to observability via distributed tracing. This utility enables seamless instrumentation of AWS Lambda functions, allowing developers to trace requests across distributed systems while maintaining Powertools for AWS Lambda commitment to developer experience.
The Powertools for AWS Lambda OpenTelemetry Tracer implements the familiar interface as the existing Powertools for AWS Lambda X-Ray tracer, allowing easy migrations while opening up integration with a broader ecosystem of observability providers through the OpenTelemetry standard.
Powertools for AWS Lambda OpenTelemetry Tracer supports three primary patterns:
Use case
Powertools for AWS Lambda customers have been asking for OpenTelemetry support to provide flexibility in their observability strategy. While AWS X-Ray offers robust tracing capabilities, many organizations have standardized on OpenTelemetry, and might use multiple observability providers that support the OpenTelemetry protocol.
The Powertools for AWS Lambda OpenTelemetry Tracer allows developers to:
Note
Interested in learning more about OpenTelemetry terms and concepts? Read this gist: https://gist.github.com/leandrodamascena/5693744f7fcf4ccf1994a7ff67a2549d
OTEL Instrumentation using Powertools for AWS Lambda
Powertools for AWS Lambda will support both auto-instrumentation, using the AWS Distro for OpenTelemetry (ADOT) Lambda Layer and manual instrumentation approaches, providing developers with flexibility while maintaining the focus on developer experience.
For auto-instrumentation, Powertools for AWS Lambda will integrate with AWS Distro for OpenTelemetry (ADOT) Lambda Layer, enabling developers to automatically instrument their Lambda functions with minimal configuration. This approach will capture traces from AWS SDK calls, HTTP requests, database operations, and other common interactions without requiring code changes.
For manual instrumentation, Powertools for AWS Lambda will provide opinionated methods that implement best practices for AWS Lambda environments, optimizing performance and configurability, while providing simplified higher level abstractions built on top of the OpenTelemetry SDK. At the same time, we'll expose the underlying OpenTelemetry SDK interfaces for power users who need direct access to the full range of OpenTelemetry capabilities.
This dual approach ensures that developers can choose the right level of observability granularity for their needs, from zero-code instrumentation to highly customized tracing, while benefiting from Powertools' Lambda-optimized defaults and simplified APIs.
Auto-Instrumentation with ADOT Lambda Layer
When using AWS Distro for OpenTelemetry (ADOT) Lambda Layer, developers benefit from a pre-packaged environment that includes the OpenTelemetry Python SDK and a pre-configured collector. This AWS Lambda Extension automatically patches all available libraries in the OpenTelemetry contrib repository, providing comprehensive instrumentation without requiring manual setup.
Powertools for AWS Lambda provides a streamlined interface to leverage this auto-instrumentation capability:
With
mode="auto"
, Powertools for AWS Lambda configures itself to work alongside AWS Distro for OpenTelemetry (ADOT) Lambda Layer auto-instrumentation capabilities, detecting its presence and optimizing the integration.Powertools for AWS Lambda provides opinionated methods like
add_span
, which wraps OpenTelemetry'sstart_as_current_span
with Lambda-optimized defaults. For instance,add_span
automatically configuresrecord_exception
to ensure exceptions are properly captured in the trace.Opinionated Auto-instrumentation
Powertools for AWS Lambda also offers opinionated configurations for common instrumentation scenarios. For example, you can easily exclude specific endpoints from being traced when using requests library:
This streamlined interface simplifies the configuration of instrumentation filters in OpenTelemetry, making it straightforward to exclude health checks, monitoring endpoints, or other high-volume, low-value traces that could increase costs.
Powertools for AWS Lambda also enhances specific instrumentations, like the
requests
library andAWS SDK
library, providing additional context and ensuring AWS Lambda-specific attributes are properly recorded. This ensures developers get better quality telemetry data without additional configuration:Manual instrumentation with Powertools for AWS Lambda
While auto-instrumentation provides a quick way to get started with tracing, manual instrumentation offers more control and customization. Powertools for AWS Lambda supports a flexible approach to manual instrumentation, allowing developers to either provide their own pre-configured OpenTelemetry components or rely on Powertools for AWS Lambda recommended defaults.
The following configurations will be supported:
Bring your own TracerProvider instance
When developers need granular control over their OpenTelemetry configuration, they can create and fully customize their own
TracerProvider
before passing it to Powertools for AWS Lambda. This approach allows developers to define every aspect of their observability pipeline - from custom resources and sampling strategies to specialized exporters and processors - while still using Powertools' Lambda-optimized instrumentation helpers:Creating a default TracerProvider with Powertools for AWS Lambda
For simple use cases, Powertools for AWS will create a default TracerProvider with Lambda-optimized settings.
The default configuration includes:
TracerProvider
with comprehensive AWS Lambda resource attributes (function name, version, runtime, etc.)BatchSpanProcessor
with parameters optimized for AWS Lambda's execution model (smaller batch sizes, faster export scheduling during shutdown)This approach enables production-ready tracing with zero configuration, while respecting AWS Lambda's environment constraints.
The default configuration includes these AWS Lambda-optimized settings:
service.name
: Defaults to the AWS Lambda function namecloud.provider
: Set to "aws"cloud.region
: Extracted from the AWS Lambda environmentcloud.account.id
: Extracted from the AWS Lambda function ARNfaas.name
: AWS Lambda Function namefaas.version
: AWS Lambda Function version/aliasfaas.instance
: AWS Lambda Instance ID for cold-start trackingfaas.max_memory
: AWS Lambda Configured memory limitmax_export_batch_size
: Set to 10 (smaller than standard defaults) to optimize for AWS Lambda's limited concurrencyschedule_delay_millis
: Set to 1000ms with more frequent flushing during AWS LambdaShutdown
to ensure traces are exported before AWS Lambda function terminationmax_queue_size
: Limited to 100 to prevent excessive memory usage in the constrained AWS Lambda environmentexport_timeout_millis
: Reduced to 3000ms to prevent blocking AWS Lambda execution for too longhttp://localhost:4317
(standard collector port within AWS Lambda when using AWS Distro for OpenTelemetry (ADOT) Lambda Layer)faas.coldstart
Setting a custom exporter with default configuration
The Powertools' Lambda-optimized TracerProvider enables developers to send their telemetry data to a specific observability provider. This hybrid approach allows developers to benefit from the default configuration, while customizing the destination for their traces - ideal for sending data in scenarios when AWS Lambda Layers/Extensions (i.e. AWS Distro for OpenTelemetry (ADOT) Lambda Layer) are not used.
Additional TracerProvider customizations
For fine-tuned control over specific aspects of your tracing configuration, you can selectively customize individual components while letting Powertools for AWS Lambda manage the rest. This targeted customization approach is ideal when you need specialized behavior - such as custom sampling logic, enhanced resource attributes, or specific trace processors - while still benefiting from Powertools for AWS Lambda optimised defaults for the remaining components:
Multiple exporters
For specific observability architectures, you can configure your tracing setup to simultaneously send telemetry data to multiple destinations. This capability enables advanced patterns such as sending production traces to your primary observability platform while routing a copy to specialized analysis tools, maintaining a local debug stream, or implementing graceful degradation paths for your telemetry pipeline:
Resource Attributes and OpenTelemetry Semantic Conventions
Powertools for AWS Lambda strictly follows OpenTelemetry's semantic conventions for all resource attributes. This ensures your telemetry data is standardized and compatible with any OpenTelemetry observability provider.
Standard Resource Attributes
When applicable, Powertools for AWS Lambda automatically populate Standard Resource Attributes, according to the OpenTelemetry specification:
cloud.provider
:"aws"
cloud.region
: AWS region of the AWS Lambda functioncloud.account.id
: AWS Account ID from the AWS Lambda Function ARNservice.name
: AWS Lambda Function Name (customizable)service.version
: AWS Lambda Function Version or Deployment IDfaas.name
: AWS Lambda Function Namefaas.version
: AWS Lambda Function Version/Aliasfaas.instance
: AWS Lambda Function Instance IDfaas.max_memory
: AWS Lambda Function Configured Memory Limittelemetry.sdk.name
:"opentelemetry"
telemetry.sdk.language
: AWS Lambda Runtime Languagetelemetry.distro.name
:"powertools-for-aws-lambda"
Custom Resource Extensions
You can extend the default Attributes while maintaining specification compliance:
Opinionated API for integration with OpenTelemetry SDK
Powertools for AWS will provide an opinionated API for OpenTelemetry while incorporating best practices for serverless environments. The opinionated methods abstract away the complexity of OpenTelemetry's lower-level APIs while providing optimised defaults for AWS Lambda functions.
capture_method
The
capture_method
decorator transforms any function or method in your code into a traced operation, by automatically creating a properly configured span. This decorator applies AWS Lambda-optimized best practices, such as exception tracking, proper span naming, and parent-child relationship:capture_lambda_handler
The
capture_lambda_handler
decorator creates a root span that encompasses the entire AWS Lambda invocation lifecycle. It intelligently integrates with the[opentelemetry-instrumentation-aws-lambda](https://pypi.org/project/opentelemetry-instrumentation-aws-lambda/)
package to capture the AWS Lambda execution context, cold-starts, and initialization phases, while automatically extracting and propagating trace context from various event sources.add_span
The
add_span
method serves as Powertools for AWS Lambda opinionated version of OpenTelemetry'sstart_as_current_span
, providing applicable default values. This method simplifies span creation, including error handling, automatic status management, and contextual attribute injection:Direct Access to OpenTelemetry SDK
For advanced observability scenarios, when fine-grained control is required, Powertools for AWS Lambda provides direct access to the underlying OpenTelemetry SDK. This escape hatch enables instrumentation patterns beyond Powertools for AWS Lambda opinionated APIs, giving developers the flexibility to implement specialized tracing solutions, while still benefiting from Powertools' Lambda-optimized configuration:
Potential Helper Functions
We're considering adding helper functions to simplify common OpenTelemetry workflows, particularly around context propagation across asynchronous boundaries. For example, we might create utilities to extract and inject trace context into JSON payloads for messaging systems like SQS, SNS, EventBridge, and other message-based services:
However, we're not yet certain if these helper functions should be part of the core implementation or added later based on community feedback.
We welcome input from the community on whether such helpers would be valuable and how they should be designed to maximize practicality while maintaining simplicity.
AWS X-Ray to OpenTelemetry migration reference
For customers migrating from AWS X-Ray to OpenTelemetry with Powertools for AWS Lambda, the following table provides a mapping between concepts and methods:
@tracer.capture_lambda_handler
@tracer.capture_lambda_handler
with tracer.capture_subsegment(name)
with tracer.add_span(name)
tracer.put_annotation(key, value)
span.set_attribute(key, value)
tracer.put_metadata(key, value)
span.set_attribute(key, value)
tracer.current_trace_id
tracer.get_current_span().get_span_context().trace_id
tracer.current_entity_id
tracer.get_current_span().get_span_context().span_id
service
parameterservice.name
resource attributetracer.patch(["boto3", "requests"])
tracer = TracerOpenTelemetry(auto_instrument=True)
faas.coldstart
tracer.ignore_endpoint(hostname, urls)
tracer.ignore_endpoint(hostname, urls)
sampler=CustomSampler()
when creating provider@tracer.capture_method
@tracer.capture_method
Evolving with the community
At Powertools for AWS Lambda, our approach is deliberately progressive. This initial implementation of OpenTelemetry tracing represents our first step into a broader integration with the OpenTelemetry ecosystem. We recognize that we may not have all the answers today, which is why community feedback will be essential in shaping our path forward.
We're looking ahead to a future where OpenTelemetry becomes a comprehensive observability foundation across all Powertools for AWS Lambda components. We anticipate developing additional RFCs for Metrics and Logging using OpenTelemetry, creating a cohesive observability experience that leverages industry standards while maintaining the developer-friendly experience Powertools for AWS lambda is known for.
We also envision integrating OpenTelemetry with other Powertools for AWS Lambda utilities. For example, a future “BatchProcessing” implementation might include built-in instrumentation:
Out of scope
To maintain focus on delivering high-quality core functionality, the following items are explicitly out of scope for this implementation:
We need your feedback
This RFC represents an initial proposal for integrating OpenTelemetry with Powertools for AWS Lambda, but we need your feedback to ensure we're building a solution that truly meets your needs.
What are we missing? Are there specific use cases we haven't addressed? How can we further improve the developer experience? Are there particular instrumentation patterns that would make your observability workflows more effective?
Please share your thoughts in this RFC, or if you prefer, via email at [email protected]
Beta Was this translation helpful? Give feedback.
All reactions