chore(logging): remove DD_CALL_BASIC_CONFIG

[Yun-Kim]

Summary

This PR removes the DD_CALL_BASIC_CONFIG environment variable and DD_LOG_FORMAT constant from the public API, and features a subtle change in the library's logging functionality to avoid configuring the user application's root-level logger. Moving forward, we will configure the ddtrace level logger instead.

Additionally, this PR adds a new default behavior to ensure that ddtrace will always log to both stdout (stream) and/or file if configured to log to file. Previously ddtrace would only log to stream OR file, but this isn't intuitive for users.

Background

DD_CALL_BASIC_CONFIG was deprecated as of #3168 (February 2022). This environment variable is currently used to enable the ddtrace library to call logging.basicConfig() to configure and format the root logger to print logs to stdout (when being run in ddtrace-run or through tracer debugging mode).

However, the ddtrace library is not a logging library and calling logging.basicConfig() will configure the root-level logger and invalidate any future calls to logging.basicConfig(), meaning this will potentially interfere with user logging configuration. User applications should be responsible for calling logging.basicConfig() instead of ddtrace.

Proposed Solution

Instead of calling loggging.basicConfig(), we instead now configure the ddtrace level logger at module startup (no longer at sitecustomize.py or tracer startup) to add a stdout Streamhandler (or file logger if specified by DD_TRACE_LOG_FILE) to avoid interfering with root-level logger configurations. Note that we still configure logging injection at sitecustomize.py and tracer startup since that requires logging to be patched.

Note: this PR leaves one usage of logging.basicConfig() in ddtrace/commands/ddtrace_run.py. This will not impact user application logging because our ddtrace-run script replaces the currently running process with the bootstrap script and user application through os.execl(...), meaning this will reset a new logging configuration to default and be unaffected once the bootstrap sitecustomize and user application starts running. This approach will leave ddtrace-run debugging logs to be printed to stdout but still avoids interfering with user application logging configurations.

Checklist

• Change(s) are motivated and described in the PR description.
• Testing strategy is described if automated tests are not included in the PR.
• Risk is outlined (performance impact, potential for breakage, maintainability, etc).
• Change is maintainable (easy to change, telemetry, documentation).
• Library release note guidelines are followed. If no release note is required, add label changelog/no-changelog.
• Documentation is included (in-code, generated user docs, public corp docs).
• Backport labels are set (if applicable)

Reviewer Checklist

• Title is accurate.
• No unnecessary changes are introduced.
• Description motivates each change.
• Avoids breaking API changes unless absolutely necessary.
• Testing strategy adequately addresses listed risk(s).
• Change is maintainable (easy to change, telemetry, documentation).
• Release note makes sense to a user of the library.
• Reviewer has explicitly acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment.
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[pr-commenter]

Benchmarks

Benchmark execution time: 2023-08-31 17:43:46

Comparing candidate commit be1315f in PR branch yunkim/remove-dd-call-basic-config with baseline commit a1b047f in branch yunkim/remove-deprecated-items.

Found 8 performance improvements and 8 performance regressions! Performance is the same for 88 metrics, 0 unstable metrics.

scenario:flasksimple-appsec-telemetry

• 🟥 max_rss_usage [+745.833KB; +944.177KB] or [+2.251%; +2.849%]

scenario:flasksimple-tracer

• 🟩 execution_time [-317.556µs; -288.028µs] or [-5.081%; -4.609%]

scenario:iastpropagation-no-propagation

• 🟥 max_rss_usage [+1.121MB; +1.247MB] or [+3.437%; +3.825%]

scenario:otelspan-add-metrics

• 🟥 max_rss_usage [+1.268MB; +1.401MB] or [+3.021%; +3.340%]

scenario:otelspan-add-tags

• 🟥 max_rss_usage [+1.201MB; +1.333MB] or [+2.857%; +3.170%]

scenario:otelspan-start-finish

• 🟥 max_rss_usage [+581.781KB; +712.146KB] or [+2.023%; +2.477%]

scenario:samplingrules-high_match

• 🟥 max_rss_usage [+581.572KB; +702.114KB] or [+2.110%; +2.547%]

scenario:sethttpmeta-obfuscation-disabled

• 🟩 max_rss_usage [-955.779KB; -825.162KB] or [-3.325%; -2.870%]

scenario:sethttpmeta-obfuscation-worst-case-explicit-query

• 🟩 max_rss_usage [-801.522KB; -652.968KB] or [-2.780%; -2.264%]

scenario:span-add-tags

• 🟥 max_rss_usage [+0.983MB; +1.161MB] or [+2.401%; +2.836%]

scenario:span-start-finish

• 🟥 max_rss_usage [+567.653KB; +724.635KB] or [+2.052%; +2.619%]

scenario:span-start-finish-telemetry

• 🟩 max_rss_usage [-775.340KB; -648.430KB] or [-2.683%; -2.244%]

scenario:span-start-finish-traceid128

• 🟩 max_rss_usage [-807.269KB; -685.723KB] or [-2.843%; -2.415%]

scenario:tracer-large

• 🟩 max_rss_usage [-921.103KB; -757.847KB] or [-3.108%; -2.557%]

scenario:tracer-medium

• 🟩 max_rss_usage [-991.529KB; -836.106KB] or [-3.430%; -2.892%]

scenario:tracer-small

• 🟩 max_rss_usage [-912.885KB; -760.741KB] or [-3.164%; -2.637%]