Skip to content

Conversation

litianningdatadog
Copy link
Contributor

@litianningdatadog litianningdatadog commented Aug 18, 2025

https://datadoghq.atlassian.net/browse/SVLS-7398

  • As part of coming release, bottlecap agent no longer launches Go-based agent when compatibility/AAP/OTLP features are active
  • Emit the same metric when detecting any of above configuration
  • Update corresponding unit tests

Manifests:

image image image

@litianningdatadog litianningdatadog requested a review from a team as a code owner August 18, 2025 21:07
@litianningdatadog litianningdatadog marked this pull request as draft August 18, 2025 22:13
@litianningdatadog litianningdatadog marked this pull request as ready for review August 21, 2025 14:48
@litianningdatadog litianningdatadog marked this pull request as draft August 21, 2025 15:04
@litianningdatadog litianningdatadog force-pushed the tianning.li/SVLS-7398 branch 2 times, most recently from fe9c5fd to a25bfcb Compare August 21, 2025 16:16
@litianningdatadog litianningdatadog marked this pull request as ready for review August 21, 2025 16:59
@duncanista duncanista requested a review from Copilot August 21, 2025 17:24
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR removes the fallback mechanism that would launch the Go-based agent when certain features (compatibility/AAP/OTLP) are enabled. Instead of falling back, the bottlecap agent now continues running and emits metrics to track these configuration scenarios.

  • Modifies config parsing to no longer return errors for compatibility/AAP/OTLP configurations
  • Adds metric emission capability to track configuration load issues with reason tags
  • Updates all test cases to reflect the new non-error behavior

Reviewed Changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.

File Description
bottlecap/src/config/mod.rs Converts fallback function from error-returning to reason-collecting, updates get_config to never fail
bottlecap/src/metrics/enhanced/lambda.rs Adds metric emission for config issues with combined tag support
bottlecap/src/metrics/enhanced/constants.rs Defines new metric constant for extension failover tracking
bottlecap/src/otlp/mod.rs Updates test cases to remove error handling expectations

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

}

fn fallback(config: &Config) -> Result<(), ConfigError> {
#[must_use]
Copy link
Preview

Copilot AI Aug 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The #[must_use] attribute on the fallback function should include a message explaining why the return value must be used, such as '#[must_use = "fallback reasons should be processed to emit appropriate metrics"]'.

Suggested change
#[must_use]
#[must_use = "fallback reasons should be processed to emit appropriate metrics"]

Copilot uses AI. Check for mistakes.

}

#[allow(clippy::module_name_repetitions)]
pub fn get_config(config_directory: &Path) -> Result<Config, ConfigError> {
#[must_use]
Copy link
Preview

Copilot AI Aug 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The #[must_use] attribute on get_config should include a message explaining why the return value must be used, such as '#[must_use = "configuration must be used to initialize the application"]'.

Suggested change
#[must_use]
#[must_use = "configuration must be used to initialize the application"]

Copilot uses AI. Check for mistakes.

}

let mut combined_tags = self.dynamic_value_tags.clone();
combined_tags.extend(additional_tags.iter().map(|(k, v)| (k.clone(), v.clone())));
Copy link
Preview

Copilot AI Aug 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cloning of keys and values in the extend operation is inefficient. Consider using references or owned values directly: combined_tags.extend(additional_tags.clone());

Suggested change
combined_tags.extend(additional_tags.iter().map(|(k, v)| (k.clone(), v.clone())));
combined_tags.extend(additional_tags.iter());

Copilot uses AI. Check for mistakes.

Comment on lines 1169 to 1170
let mut metrics_flusher = MetricsFlusher::new(flusher_config);
let _ = metrics_flusher.flush().await;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want to create a flusher here, why not reuse the existing one and let the flushing strategies handle this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acked

@litianningdatadog litianningdatadog force-pushed the tianning.li/SVLS-7398 branch 4 times, most recently from 72392c1 to bd05b0e Compare August 21, 2025 22:10
Comment on lines 1129 to 1130
let lambda_enhanced_metrics =
enhanced_metrics::new(Arc::clone(metrics_aggr), Arc::clone(config));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we reuse this too? Create it, use it, then send it to the lifecycle processor? That way we don't allocate twice?

You won't get the right amount of tags after the invocation has occurred, like runtime

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acked

Copy link
Contributor

@duncanista duncanista left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one last comment and we should be good to go I think!

Copy link
Contributor

@duncanista duncanista left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM – let's not merge until the next-next release, as we want this to be released on v86

let config = match config::get_config(Path::new(&lambda_directory)) {
Ok(config) => Arc::new(config),
Err(_e) => {
let err = Command::new("/opt/datadog-agent-go").exec();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally!

}
};

let config = Arc::new(config::get_config(Path::new(&lambda_directory)));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe not for this release, but does this mean we can defer config parsing until after extension registration now? Will that save us any MS?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I doubt it based on my perception. At least the extension setup still relies on the configuration such as tracing log and FIPS.

- No error is thrown in case fallback (pending)
- More debugging logs (will be removed before official code merge)
@litianningdatadog litianningdatadog merged commit 0f59845 into main Aug 28, 2025
46 checks passed
@litianningdatadog litianningdatadog deleted the tianning.li/SVLS-7398 branch August 28, 2025 18:59
duncanista added a commit that referenced this pull request Aug 28, 2025
duncanista pushed a commit that referenced this pull request Aug 29, 2025
…ig (#788)

https://datadoghq.atlassian.net/browse/SVLS-7398

- As part of coming release, bottlecap agent no longer launches Go-based
agent when compatibility/AAP/OTLP features are active
- Emit the same metric when detecting any of above configuration
- Update corresponding unit tests

Manifests:
- [Test lambda
function](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/ltn1-fullinstrument-bn-cold-python310-lambda?code=&subtab=envVars&tab=testing)
with
[logs](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn1-fullinstrument-bn-cold-python310-lambda/log-events/2025$252F08$252F21$252F$255B$2524LATEST$255Df3788d359677452dad162488ff15456f$3FfilterPattern$3Dotel)
showing compatibility/AAP/OTPL are enabled
<img width="2260" height="454" alt="image"
src="https://github.com/user-attachments/assets/5dfd4954-5191-4390-83f5-a8eb3bffb9d3"
/>

-
[Logging](https://app.datadoghq.com/logs/livetail?query=functionname%3Altn1-fullinstrument-bn-cold-python310-lambda%20Metric&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=driveline&stream_sort=desc&viz=stream&from_ts=1755787655569&to_ts=1755787689060&live=false)
<img width="1058" height="911" alt="image"
src="https://github.com/user-attachments/assets/629f75d1-e115-4478-afac-ad16d9369fa7"
/>

-
[Metric](https://app.datadoghq.com/screen/integration/aws_lambda_enhanced_metrics?fromUser=false&fullscreen_end_ts=1755788220000&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1755787200000&fullscreen_widget=2&graph-explorer__tile_def=N4IgbglgXiBcIBcD2AHANhAzgkAaEAxgK7ZIC2A%2BhgHYDWmcA2gLr4BOApgI5EfYOxGoTphRJqmDhQBmSNmQCGOeJgIK0CtnhA8ObCHyagAJkoUVMSImwIc4IMhwT6CDfNQWP7utgE8AjNo%2BvvaYRGSwpggKxkgA5gB0kmxgemh8mAkcAB4IHBIQ4gnSChBoSKlswAAkCgDumBQKBARW1Ai41ZxxhdSd0kTUBAi9AL4ABABGvuPAA0Mj4h6OowkKja2DCAAUAJTaCnFx3UpyoeEgo6wgsvJEGgJCN3Jk9wrevH6BV-iWbMqgTbtOAAJgADPg5MY9BRpkZEL4UHZ4LdXhptBBqNDsnAISAoXp7NDVJdmKMfiBsL50nBgOSgA&refresh_mode=sliding&from_ts=1755783890661&to_ts=1755787490661&live=true)
<img width="1227" height="1196" alt="image"
src="https://github.com/user-attachments/assets/2922eb54-9853-4512-a902-dfa97916b643"
/>
litianningdatadog added a commit that referenced this pull request Aug 29, 2025
…ig (#788)

https://datadoghq.atlassian.net/browse/SVLS-7398

- As part of coming release, bottlecap agent no longer launches Go-based
agent when compatibility/AAP/OTLP features are active
- Emit the same metric when detecting any of above configuration
- Update corresponding unit tests

Manifests:
- [Test lambda
function](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/ltn1-fullinstrument-bn-cold-python310-lambda?code=&subtab=envVars&tab=testing)
with
[logs](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn1-fullinstrument-bn-cold-python310-lambda/log-events/2025$252F08$252F21$252F$255B$2524LATEST$255Df3788d359677452dad162488ff15456f$3FfilterPattern$3Dotel)
showing compatibility/AAP/OTPL are enabled
<img width="2260" height="454" alt="image"
src="https://github.com/user-attachments/assets/5dfd4954-5191-4390-83f5-a8eb3bffb9d3"
/>

-
[Logging](https://app.datadoghq.com/logs/livetail?query=functionname%3Altn1-fullinstrument-bn-cold-python310-lambda%20Metric&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=driveline&stream_sort=desc&viz=stream&from_ts=1755787655569&to_ts=1755787689060&live=false)
<img width="1058" height="911" alt="image"
src="https://github.com/user-attachments/assets/629f75d1-e115-4478-afac-ad16d9369fa7"
/>

-
[Metric](https://app.datadoghq.com/screen/integration/aws_lambda_enhanced_metrics?fromUser=false&fullscreen_end_ts=1755788220000&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1755787200000&fullscreen_widget=2&graph-explorer__tile_def=N4IgbglgXiBcIBcD2AHANhAzgkAaEAxgK7ZIC2A%2BhgHYDWmcA2gLr4BOApgI5EfYOxGoTphRJqmDhQBmSNmQCGOeJgIK0CtnhA8ObCHyagAJkoUVMSImwIc4IMhwT6CDfNQWP7utgE8AjNo%2BvvaYRGSwpggKxkgA5gB0kmxgemh8mAkcAB4IHBIQ4gnSChBoSKlswAAkCgDumBQKBARW1Ai41ZxxhdSd0kTUBAi9AL4ABABGvuPAA0Mj4h6OowkKja2DCAAUAJTaCnFx3UpyoeEgo6wgsvJEGgJCN3Jk9wrevH6BV-iWbMqgTbtOAAJgADPg5MY9BRpkZEL4UHZ4LdXhptBBqNDsnAISAoXp7NDVJdmKMfiBsL50nBgOSgA&refresh_mode=sliding&from_ts=1755783890661&to_ts=1755787490661&live=true)
<img width="1227" height="1196" alt="image"
src="https://github.com/user-attachments/assets/2922eb54-9853-4512-a902-dfa97916b643"
/>
litianningdatadog added a commit that referenced this pull request Aug 29, 2025
…ig (#788)

https://datadoghq.atlassian.net/browse/SVLS-7398

- As part of coming release, bottlecap agent no longer launches Go-based
agent when compatibility/AAP/OTLP features are active
- Emit the same metric when detecting any of above configuration
- Update corresponding unit tests

Manifests:
- [Test lambda
function](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/ltn1-fullinstrument-bn-cold-python310-lambda?code=&subtab=envVars&tab=testing)
with
[logs](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn1-fullinstrument-bn-cold-python310-lambda/log-events/2025$252F08$252F21$252F$255B$2524LATEST$255Df3788d359677452dad162488ff15456f$3FfilterPattern$3Dotel)
showing compatibility/AAP/OTPL are enabled
<img width="2260" height="454" alt="image"
src="https://github.com/user-attachments/assets/5dfd4954-5191-4390-83f5-a8eb3bffb9d3"
/>

-
[Logging](https://app.datadoghq.com/logs/livetail?query=functionname%3Altn1-fullinstrument-bn-cold-python310-lambda%20Metric&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=driveline&stream_sort=desc&viz=stream&from_ts=1755787655569&to_ts=1755787689060&live=false)
<img width="1058" height="911" alt="image"
src="https://github.com/user-attachments/assets/629f75d1-e115-4478-afac-ad16d9369fa7"
/>

-
[Metric](https://app.datadoghq.com/screen/integration/aws_lambda_enhanced_metrics?fromUser=false&fullscreen_end_ts=1755788220000&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1755787200000&fullscreen_widget=2&graph-explorer__tile_def=N4IgbglgXiBcIBcD2AHANhAzgkAaEAxgK7ZIC2A%2BhgHYDWmcA2gLr4BOApgI5EfYOxGoTphRJqmDhQBmSNmQCGOeJgIK0CtnhA8ObCHyagAJkoUVMSImwIc4IMhwT6CDfNQWP7utgE8AjNo%2BvvaYRGSwpggKxkgA5gB0kmxgemh8mAkcAB4IHBIQ4gnSChBoSKlswAAkCgDumBQKBARW1Ai41ZxxhdSd0kTUBAi9AL4ABABGvuPAA0Mj4h6OowkKja2DCAAUAJTaCnFx3UpyoeEgo6wgsvJEGgJCN3Jk9wrevH6BV-iWbMqgTbtOAAJgADPg5MY9BRpkZEL4UHZ4LdXhptBBqNDsnAISAoXp7NDVJdmKMfiBsL50nBgOSgA&refresh_mode=sliding&from_ts=1755783890661&to_ts=1755787490661&live=true)
<img width="1227" height="1196" alt="image"
src="https://github.com/user-attachments/assets/2922eb54-9853-4512-a902-dfa97916b643"
/>
litianningdatadog added a commit that referenced this pull request Aug 29, 2025
…ig (#788)

https://datadoghq.atlassian.net/browse/SVLS-7398

- As part of coming release, bottlecap agent no longer launches Go-based
agent when compatibility/AAP/OTLP features are active
- Emit the same metric when detecting any of above configuration
- Update corresponding unit tests

Manifests:
- [Test lambda
function](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/ltn1-fullinstrument-bn-cold-python310-lambda?code=&subtab=envVars&tab=testing)
with
[logs](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn1-fullinstrument-bn-cold-python310-lambda/log-events/2025$252F08$252F21$252F$255B$2524LATEST$255Df3788d359677452dad162488ff15456f$3FfilterPattern$3Dotel)
showing compatibility/AAP/OTPL are enabled
<img width="2260" height="454" alt="image"
src="https://github.com/user-attachments/assets/5dfd4954-5191-4390-83f5-a8eb3bffb9d3"
/>

-
[Logging](https://app.datadoghq.com/logs/livetail?query=functionname%3Altn1-fullinstrument-bn-cold-python310-lambda%20Metric&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=driveline&stream_sort=desc&viz=stream&from_ts=1755787655569&to_ts=1755787689060&live=false)
<img width="1058" height="911" alt="image"
src="https://github.com/user-attachments/assets/629f75d1-e115-4478-afac-ad16d9369fa7"
/>

-
[Metric](https://app.datadoghq.com/screen/integration/aws_lambda_enhanced_metrics?fromUser=false&fullscreen_end_ts=1755788220000&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1755787200000&fullscreen_widget=2&graph-explorer__tile_def=N4IgbglgXiBcIBcD2AHANhAzgkAaEAxgK7ZIC2A%2BhgHYDWmcA2gLr4BOApgI5EfYOxGoTphRJqmDhQBmSNmQCGOeJgIK0CtnhA8ObCHyagAJkoUVMSImwIc4IMhwT6CDfNQWP7utgE8AjNo%2BvvaYRGSwpggKxkgA5gB0kmxgemh8mAkcAB4IHBIQ4gnSChBoSKlswAAkCgDumBQKBARW1Ai41ZxxhdSd0kTUBAi9AL4ABABGvuPAA0Mj4h6OowkKja2DCAAUAJTaCnFx3UpyoeEgo6wgsvJEGgJCN3Jk9wrevH6BV-iWbMqgTbtOAAJgADPg5MY9BRpkZEL4UHZ4LdXhptBBqNDsnAISAoXp7NDVJdmKMfiBsL50nBgOSgA&refresh_mode=sliding&from_ts=1755783890661&to_ts=1755787490661&live=true)
<img width="1227" height="1196" alt="image"
src="https://github.com/user-attachments/assets/2922eb54-9853-4512-a902-dfa97916b643"
/>
litianningdatadog added a commit that referenced this pull request Sep 5, 2025
…ig (#788)

https://datadoghq.atlassian.net/browse/SVLS-7398

- As part of coming release, bottlecap agent no longer launches Go-based
agent when compatibility/AAP/OTLP features are active
- Emit the same metric when detecting any of above configuration
- Update corresponding unit tests

Manifests:
- [Test lambda
function](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/ltn1-fullinstrument-bn-cold-python310-lambda?code=&subtab=envVars&tab=testing)
with
[logs](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn1-fullinstrument-bn-cold-python310-lambda/log-events/2025$252F08$252F21$252F$255B$2524LATEST$255Df3788d359677452dad162488ff15456f$3FfilterPattern$3Dotel)
showing compatibility/AAP/OTPL are enabled
<img width="2260" height="454" alt="image"
src="https://github.com/user-attachments/assets/5dfd4954-5191-4390-83f5-a8eb3bffb9d3"
/>

-
[Logging](https://app.datadoghq.com/logs/livetail?query=functionname%3Altn1-fullinstrument-bn-cold-python310-lambda%20Metric&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=driveline&stream_sort=desc&viz=stream&from_ts=1755787655569&to_ts=1755787689060&live=false)
<img width="1058" height="911" alt="image"
src="https://github.com/user-attachments/assets/629f75d1-e115-4478-afac-ad16d9369fa7"
/>

-
[Metric](https://app.datadoghq.com/screen/integration/aws_lambda_enhanced_metrics?fromUser=false&fullscreen_end_ts=1755788220000&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1755787200000&fullscreen_widget=2&graph-explorer__tile_def=N4IgbglgXiBcIBcD2AHANhAzgkAaEAxgK7ZIC2A%2BhgHYDWmcA2gLr4BOApgI5EfYOxGoTphRJqmDhQBmSNmQCGOeJgIK0CtnhA8ObCHyagAJkoUVMSImwIc4IMhwT6CDfNQWP7utgE8AjNo%2BvvaYRGSwpggKxkgA5gB0kmxgemh8mAkcAB4IHBIQ4gnSChBoSKlswAAkCgDumBQKBARW1Ai41ZxxhdSd0kTUBAi9AL4ABABGvuPAA0Mj4h6OowkKja2DCAAUAJTaCnFx3UpyoeEgo6wgsvJEGgJCN3Jk9wrevH6BV-iWbMqgTbtOAAJgADPg5MY9BRpkZEL4UHZ4LdXhptBBqNDsnAISAoXp7NDVJdmKMfiBsL50nBgOSgA&refresh_mode=sliding&from_ts=1755783890661&to_ts=1755787490661&live=true)
<img width="1227" height="1196" alt="image"
src="https://github.com/user-attachments/assets/2922eb54-9853-4512-a902-dfa97916b643"
/>
litianningdatadog added a commit that referenced this pull request Sep 5, 2025
…ig (#788)

https://datadoghq.atlassian.net/browse/SVLS-7398

- As part of coming release, bottlecap agent no longer launches Go-based
agent when compatibility/AAP/OTLP features are active
- Emit the same metric when detecting any of above configuration
- Update corresponding unit tests
- Refactor the references of enhanced metrics.

Manifests:
- [Test lambda
function](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/ltn1-fullinstrument-bn-cold-python310-lambda?code=&subtab=envVars&tab=testing)
with
[logs](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn1-fullinstrument-bn-cold-python310-lambda/log-events/2025$252F08$252F21$252F$255B$2524LATEST$255Df3788d359677452dad162488ff15456f$3FfilterPattern$3Dotel)
showing compatibility/AAP/OTPL are enabled
<img width="2260" height="454" alt="image"
src="https://github.com/user-attachments/assets/5dfd4954-5191-4390-83f5-a8eb3bffb9d3"
/>

-
[Logging](https://app.datadoghq.com/logs/livetail?query=functionname%3Altn1-fullinstrument-bn-cold-python310-lambda%20Metric&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=driveline&stream_sort=desc&viz=stream&from_ts=1755787655569&to_ts=1755787689060&live=false)
<img width="1058" height="911" alt="image"
src="https://github.com/user-attachments/assets/629f75d1-e115-4478-afac-ad16d9369fa7"
/>

-
[Metric](https://app.datadoghq.com/screen/integration/aws_lambda_enhanced_metrics?fromUser=false&fullscreen_end_ts=1755788220000&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1755787200000&fullscreen_widget=2&graph-explorer__tile_def=N4IgbglgXiBcIBcD2AHANhAzgkAaEAxgK7ZIC2A%2BhgHYDWmcA2gLr4BOApgI5EfYOxGoTphRJqmDhQBmSNmQCGOeJgIK0CtnhA8ObCHyagAJkoUVMSImwIc4IMhwT6CDfNQWP7utgE8AjNo%2BvvaYRGSwpggKxkgA5gB0kmxgemh8mAkcAB4IHBIQ4gnSChBoSKlswAAkCgDumBQKBARW1Ai41ZxxhdSd0kTUBAi9AL4ABABGvuPAA0Mj4h6OowkKja2DCAAUAJTaCnFx3UpyoeEgo6wgsvJEGgJCN3Jk9wrevH6BV-iWbMqgTbtOAAJgADPg5MY9BRpkZEL4UHZ4LdXhptBBqNDsnAISAoXp7NDVJdmKMfiBsL50nBgOSgA&refresh_mode=sliding&from_ts=1755783890661&to_ts=1755787490661&live=true)
<img width="1227" height="1196" alt="image"
src="https://github.com/user-attachments/assets/2922eb54-9853-4512-a902-dfa97916b643"
/>
litianningdatadog added a commit that referenced this pull request Sep 8, 2025
…ig (#788)

https://datadoghq.atlassian.net/browse/SVLS-7398

- As part of coming release, bottlecap agent no longer launches Go-based
agent when compatibility/AAP/OTLP features are active
- Emit the same metric when detecting any of above configuration
- Update corresponding unit tests

Manifests:
- [Test lambda
function](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/ltn1-fullinstrument-bn-cold-python310-lambda?code=&subtab=envVars&tab=testing)
with
[logs](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn1-fullinstrument-bn-cold-python310-lambda/log-events/2025$252F08$252F21$252F$255B$2524LATEST$255Df3788d359677452dad162488ff15456f$3FfilterPattern$3Dotel)
showing compatibility/AAP/OTPL are enabled
<img width="2260" height="454" alt="image"
src="https://github.com/user-attachments/assets/5dfd4954-5191-4390-83f5-a8eb3bffb9d3"
/>

-
[Logging](https://app.datadoghq.com/logs/livetail?query=functionname%3Altn1-fullinstrument-bn-cold-python310-lambda%20Metric&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=driveline&stream_sort=desc&viz=stream&from_ts=1755787655569&to_ts=1755787689060&live=false)
<img width="1058" height="911" alt="image"
src="https://github.com/user-attachments/assets/629f75d1-e115-4478-afac-ad16d9369fa7"
/>

-
[Metric](https://app.datadoghq.com/screen/integration/aws_lambda_enhanced_metrics?fromUser=false&fullscreen_end_ts=1755788220000&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1755787200000&fullscreen_widget=2&graph-explorer__tile_def=N4IgbglgXiBcIBcD2AHANhAzgkAaEAxgK7ZIC2A%2BhgHYDWmcA2gLr4BOApgI5EfYOxGoTphRJqmDhQBmSNmQCGOeJgIK0CtnhA8ObCHyagAJkoUVMSImwIc4IMhwT6CDfNQWP7utgE8AjNo%2BvvaYRGSwpggKxkgA5gB0kmxgemh8mAkcAB4IHBIQ4gnSChBoSKlswAAkCgDumBQKBARW1Ai41ZxxhdSd0kTUBAi9AL4ABABGvuPAA0Mj4h6OowkKja2DCAAUAJTaCnFx3UpyoeEgo6wgsvJEGgJCN3Jk9wrevH6BV-iWbMqgTbtOAAJgADPg5MY9BRpkZEL4UHZ4LdXhptBBqNDsnAISAoXp7NDVJdmKMfiBsL50nBgOSgA&refresh_mode=sliding&from_ts=1755783890661&to_ts=1755787490661&live=true)
<img width="1227" height="1196" alt="image"
src="https://github.com/user-attachments/assets/2922eb54-9853-4512-a902-dfa97916b643"
/>
litianningdatadog added a commit that referenced this pull request Sep 10, 2025
…ig (#788)

https://datadoghq.atlassian.net/browse/SVLS-7398

- As part of coming release, bottlecap agent no longer launches Go-based
agent when compatibility/AAP/OTLP features are active
- Emit the same metric when detecting any of above configuration
- Update corresponding unit tests
- Refactor the references of enhanced metrics.

Manifests:
- [Test lambda
function](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/ltn1-fullinstrument-bn-cold-python310-lambda?code=&subtab=envVars&tab=testing)
with
[logs](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn1-fullinstrument-bn-cold-python310-lambda/log-events/2025$252F08$252F21$252F$255B$2524LATEST$255Df3788d359677452dad162488ff15456f$3FfilterPattern$3Dotel)
showing compatibility/AAP/OTPL are enabled
<img width="2260" height="454" alt="image"
src="https://github.com/user-attachments/assets/5dfd4954-5191-4390-83f5-a8eb3bffb9d3"
/>

-
[Logging](https://app.datadoghq.com/logs/livetail?query=functionname%3Altn1-fullinstrument-bn-cold-python310-lambda%20Metric&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=driveline&stream_sort=desc&viz=stream&from_ts=1755787655569&to_ts=1755787689060&live=false)
<img width="1058" height="911" alt="image"
src="https://github.com/user-attachments/assets/629f75d1-e115-4478-afac-ad16d9369fa7"
/>

-
[Metric](https://app.datadoghq.com/screen/integration/aws_lambda_enhanced_metrics?fromUser=false&fullscreen_end_ts=1755788220000&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1755787200000&fullscreen_widget=2&graph-explorer__tile_def=N4IgbglgXiBcIBcD2AHANhAzgkAaEAxgK7ZIC2A%2BhgHYDWmcA2gLr4BOApgI5EfYOxGoTphRJqmDhQBmSNmQCGOeJgIK0CtnhA8ObCHyagAJkoUVMSImwIc4IMhwT6CDfNQWP7utgE8AjNo%2BvvaYRGSwpggKxkgA5gB0kmxgemh8mAkcAB4IHBIQ4gnSChBoSKlswAAkCgDumBQKBARW1Ai41ZxxhdSd0kTUBAi9AL4ABABGvuPAA0Mj4h6OowkKja2DCAAUAJTaCnFx3UpyoeEgo6wgsvJEGgJCN3Jk9wrevH6BV-iWbMqgTbtOAAJgADPg5MY9BRpkZEL4UHZ4LdXhptBBqNDsnAISAoXp7NDVJdmKMfiBsL50nBgOSgA&refresh_mode=sliding&from_ts=1755783890661&to_ts=1755787490661&live=true)
<img width="1227" height="1196" alt="image"
src="https://github.com/user-attachments/assets/2922eb54-9853-4512-a902-dfa97916b643"
/>
litianningdatadog added a commit that referenced this pull request Sep 10, 2025
…ig (#788)

https://datadoghq.atlassian.net/browse/SVLS-7398

- As part of coming release, bottlecap agent no longer launches Go-based
agent when compatibility/AAP/OTLP features are active
- Emit the same metric when detecting any of above configuration
- Update corresponding unit tests
- Refactor the references of enhanced metrics.

Manifests:
- [Test lambda
function](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/ltn1-fullinstrument-bn-cold-python310-lambda?code=&subtab=envVars&tab=testing)
with
[logs](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn1-fullinstrument-bn-cold-python310-lambda/log-events/2025$252F08$252F21$252F$255B$2524LATEST$255Df3788d359677452dad162488ff15456f$3FfilterPattern$3Dotel)
showing compatibility/AAP/OTPL are enabled
<img width="2260" height="454" alt="image"
src="https://github.com/user-attachments/assets/5dfd4954-5191-4390-83f5-a8eb3bffb9d3"
/>

-
[Logging](https://app.datadoghq.com/logs/livetail?query=functionname%3Altn1-fullinstrument-bn-cold-python310-lambda%20Metric&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=driveline&stream_sort=desc&viz=stream&from_ts=1755787655569&to_ts=1755787689060&live=false)
<img width="1058" height="911" alt="image"
src="https://github.com/user-attachments/assets/629f75d1-e115-4478-afac-ad16d9369fa7"
/>

-
[Metric](https://app.datadoghq.com/screen/integration/aws_lambda_enhanced_metrics?fromUser=false&fullscreen_end_ts=1755788220000&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1755787200000&fullscreen_widget=2&graph-explorer__tile_def=N4IgbglgXiBcIBcD2AHANhAzgkAaEAxgK7ZIC2A%2BhgHYDWmcA2gLr4BOApgI5EfYOxGoTphRJqmDhQBmSNmQCGOeJgIK0CtnhA8ObCHyagAJkoUVMSImwIc4IMhwT6CDfNQWP7utgE8AjNo%2BvvaYRGSwpggKxkgA5gB0kmxgemh8mAkcAB4IHBIQ4gnSChBoSKlswAAkCgDumBQKBARW1Ai41ZxxhdSd0kTUBAi9AL4ABABGvuPAA0Mj4h6OowkKja2DCAAUAJTaCnFx3UpyoeEgo6wgsvJEGgJCN3Jk9wrevH6BV-iWbMqgTbtOAAJgADPg5MY9BRpkZEL4UHZ4LdXhptBBqNDsnAISAoXp7NDVJdmKMfiBsL50nBgOSgA&refresh_mode=sliding&from_ts=1755783890661&to_ts=1755787490661&live=true)
<img width="1227" height="1196" alt="image"
src="https://github.com/user-attachments/assets/2922eb54-9853-4512-a902-dfa97916b643"
/>
litianningdatadog added a commit that referenced this pull request Sep 11, 2025
…AP config (#817)

Cherry pick of previously reverted #788 

https://datadoghq.atlassian.net/browse/SVLS-7398

- As part of coming release, bottlecap agent no longer launches Go-based
agent when compatibility/AAP/OTLP features are active
- Emit the same metric when detecting any of above configuration
- Update corresponding unit tests

Attention: it is an known issue with .Net
aws/aws-lambda-dotnet#2093

Manifests:
- [Test lambda
function](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/ltn1-fullinstrument-bn-cold-python310-lambda?code=&subtab=envVars&tab=testing)
with

[logs](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn1-fullinstrument-bn-cold-python310-lambda/log-events/2025$252F08$252F21$252F$255B$2524LATEST$255Df3788d359677452dad162488ff15456f$3FfilterPattern$3Dotel)
showing compatibility/AAP/OTPL are enabled
<img width="2260" height="454" alt="image"

src="https://github.com/user-attachments/assets/5dfd4954-5191-4390-83f5-a8eb3bffb9d3"
/>

-

[Logging](https://app.datadoghq.com/logs/livetail?query=functionname%3Altn1-fullinstrument-bn-cold-python310-lambda%20Metric&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=driveline&stream_sort=desc&viz=stream&from_ts=1755787655569&to_ts=1755787689060&live=false)
<img width="1058" height="911" alt="image"

src="https://github.com/user-attachments/assets/629f75d1-e115-4478-afac-ad16d9369fa7"
/>

-

[Metric](https://app.datadoghq.com/screen/integration/aws_lambda_enhanced_metrics?fromUser=false&fullscreen_end_ts=1755788220000&fullscreen_paused=true&fullscreen_refresh_mode=paused&fullscreen_section=overview&fullscreen_start_ts=1755787200000&fullscreen_widget=2&graph-explorer__tile_def=N4IgbglgXiBcIBcD2AHANhAzgkAaEAxgK7ZIC2A%2BhgHYDWmcA2gLr4BOApgI5EfYOxGoTphRJqmDhQBmSNmQCGOeJgIK0CtnhA8ObCHyagAJkoUVMSImwIc4IMhwT6CDfNQWP7utgE8AjNo%2BvvaYRGSwpggKxkgA5gB0kmxgemh8mAkcAB4IHBIQ4gnSChBoSKlswAAkCgDumBQKBARW1Ai41ZxxhdSd0kTUBAi9AL4ABABGvuPAA0Mj4h6OowkKja2DCAAUAJTaCnFx3UpyoeEgo6wgsvJEGgJCN3Jk9wrevH6BV-iWbMqgTbtOAAJgADPg5MY9BRpkZEL4UHZ4LdXhptBBqNDsnAISAoXp7NDVJdmKMfiBsL50nBgOSgA&refresh_mode=sliding&from_ts=1755783890661&to_ts=1755787490661&live=true)
<img width="1227" height="1196" alt="image"

src="https://github.com/user-attachments/assets/2922eb54-9853-4512-a902-dfa97916b643"
/>
====
Another manifest for .Net:
- [Lambda
function](https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/ltn1-fullinstrument-bn-cold-dotnet6-lambda?code=&subtab=envVars&tab=testing)
-
[Log](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/$252Faws$252Flambda$252Fltn1-fullinstrument-bn-cold-dotnet6-lambda/log-events/2025$252F08$252F29$252F$255B$2524LATEST$255D15ca867ee94049129ed461283ae46f01$3FfilterPattern$3Dfailover)
- Configuration
<img width="1490" height="902" alt="image"
src="https://github.com/user-attachments/assets/b070e5e1-8335-4494-877f-6475d9959af2"
/>
- Log shows the issue reasons
<img width="990" height="536" alt="image"
src="https://github.com/user-attachments/assets/5503de33-ea92-401c-a595-c165e39b0c6e"
/>
<img width="848" height="410" alt="image"
src="https://github.com/user-attachments/assets/54d1e87c-93e7-4084-8a9a-173cb7d0c4a7"
/>
<img width="938" height="458" alt="image"
src="https://github.com/user-attachments/assets/4f205ec2-d923-47d1-9005-762650798894"
/>

---------

Co-authored-by: Tianning Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants