[BUG]: FastAPI spans no longer appearing #15250
Description
Tracer Version(s)
3.17.2,3.18.0
Python Version(s)
3.11
Pip Version(s)
24.0
Bug Report
After upgrading to v 3.x yesterday we noticed we were no longer ingesting any fastapi.request trace metrics, but would see occasional aiohttp.request metrics.
We are running our service in Cloud Run, using gunicorn and fastapi.
After a bit of binary searching I found ddtrace 3.17.1 worked normally but 3.17.2 and 3.18.0 failed.
I am suspicious of #15018 because that's one of the few relevant commits in 3.17.2 and I can see that it is involved in generating fastapi.request spans here:
.Reproduction Code
The problem seems to repro with a simple health-check handler like this.
app = FastAPI(title="Test")
@app.get("/health", status_code=status.HTTP_204_NO_CONTENT)
async def health(request: Request) -> None:
...
Error Logs
I didn't see any error logs. Some ddtrace + agent logs are below:
INFO 2025-11-13T10:09:54.682358Z Request GET http://127.0.0.1/health responded with 204
INFO 2025-11-13T10:09:54.682419Z 169.254.169.126:4149 - "GET /health HTTP/1.1" 204
DEFAULT 2025-11-13T10:09:54.682978Z Sampling decision applied to <Span(id=6846294105390117395,trace_id=139681515962726960068732605767401350414,parent_id=None,name=fastapi.request)>: sampled=True sample_rate=1.0 sampling_mechanism=0 matched_trace_sampling_rule=None agent_sampled=None rules=[] sampler_id=139671392768768
DEBUG 2025-11-13T10:09:54.683015Z Sampling decision applied to <Span(id=6846294105390117395,trace_id=139681515962726960068732605767401350414,parent_id=None,name=fastapi.request)>: sampled=True sample_rate=1.0 sampling_mechanism=0 matched_trace_sampling_rule=None agent_sampled=None rules=[] sampler_id=139671392768768
DEFAULT 2025-11-13T10:09:54.683052Z Encoding 6 spans. Spans processed: 6. Spans dropped by trace processors: 0. Unfinished spans remaining in the span aggregator: 0. (trace_id: 139681515962726960068732605767401350414) (top level span: name=fastapi.request) (sampling_priority: 1) (sampling_mechanism: -0) (partial flush triggered: False)
DEBUG 2025-11-13T10:09:54.683107Z Encoding 6 spans. Spans processed: 6. Spans dropped by trace processors: 0. Unfinished spans remaining in the span aggregator: 0. (trace_id: 139681515962726960068732605767401350414) (top level span: name=fastapi.request) (sampling_priority: 1) (sampling_mechanism: -0) (partial flush triggered: False)
DEFAULT 2025-11-13T10:09:54.683256Z finishing span - Span(name='fastapi.request', span_id=6846294105390117395, parent_id=None, trace_id=139681515962726960068732605767401350414, service='fastapi', resource='GET /health', type='web', start=1763028594671723497, end=1763028594680696604, duration=8973107, error=0, tags={'runtime-id': '983ba21f294c496b8aa9eff130691c32', 'env': 'dev', 'component': 'fastapi', 'span.kind': 'server', 'http.method': 'GET', 'http.url': 'http://127.0.0.1/health', 'http.version': '1.1', 'asgi.version': '3.0', 'asgi.spec_version': '2.3', 'http.route': '/health', 'http.status_code': '204', '_dd.base_service': 'image-checker', '_dd.p.dm': '-0', '_dd.git.repository_url': 'https://github.com/', '_dd.git.commit.sha': 'af5d7c411d40d8a25e85fdd251f28929b1c4a02f', 'language': 'python', '_dd.p.tid': '6915ae7200000000'}, metrics={'process_id': 23, '_dd.top_level': 1, '_sampling_priority_v1': 1, '_dd.tracer_kr': 1.0}, links=[], events=[], context=Context(trace_id=139681515962726960068732605767401350414, span_id=6846294105390117395, _meta={'_dd.p.dm': '-0'}, _metrics={'_sampling_priority_v1': 1}, _span_links=[], _baggage={}, _is_remote=False), service_entry_span_name=fastapi.request) (enabled:True)
DEBUG 2025-11-13T10:09:54.683461Z finishing span - Span(name='fastapi.request', span_id=6846294105390117395, parent_id=None, trace_id=139681515962726960068732605767401350414, service='fastapi', resource='GET /health', type='web', start=1763028594671723497, end=1763028594680696604, duration=8973107, error=0, tags={'runtime-id': '983ba21f294c496b8aa9eff130691c32', 'env': 'dev', 'component': 'fastapi', 'span.kind': 'server', 'http.method': 'GET', 'http.url': 'http://127.0.0.1/health', 'http.version': '1.1', 'asgi.version': '3.0', 'asgi.spec_version': '2.3', 'http.route': '/health', 'http.status_code': '204', '_dd.base_service': 'image-checker', '_dd.p.dm': '-0', '_dd.git.repository_url': 'https://github.com/', '_dd.git.commit.sha': 'af5d7c411d40d8a25e85fdd251f28929b1c4a02f', 'language': 'python', '_dd.p.tid': '6915ae7200000000'}, metrics={'process_id': 23, '_dd.top_level': 1, '_sampling_priority_v1': 1, '_dd.tracer_kr': 1.0}, links=[], events=[], context=Context(trace_id=139681515962726960068732605767401350414, span_id=6846294105390117395, _meta={'_dd.p.dm': '-0'}, _metrics={'_sampling_priority_v1': 1}, _span_links=[], _baggage={}, _is_remote=False), service_entry_span_name=fastapi.request) (enabled:True)
DEFAULT 2025-11-13T10:09:55.384122Z 2025-11-13 10:09:55 UTC | SERVERLESS_INIT | DEBUG | Received a Flush trigger
DEFAULT 2025-11-13T10:09:55.584144Z 2025-11-13 10:09:55 UTC | SERVERLESS_INIT | DEBUG | Demultiplexer: sendIterableSeries: start sending iterable series to the serializer
DEFAULT 2025-11-13T10:09:56.582121Z 2025-11-13 10:09:56 UTC | SERVERLESS_INIT | DEBUG | The payload was not too big, returning the full payload
DEFAULT 2025-11-13T10:09:56.781620Z 2025-11-13 10:09:56 UTC | SERVERLESS_INIT | DEBUG | SyncForwarder has flushed 1 transactions
DEFAULT 2025-11-13T10:09:56.781636Z 2025-11-13 10:09:56 UTC | SERVERLESS_INIT | DEBUG | Demultiplexer: sendIterableSeries: stop routine
DEFAULT 2025-11-13T10:09:57.181829Z 2025-11-13 10:09:57 UTC | SERVERLESS_INIT | DEBUG | Non-boolean value %s found in header %s, defaulting to true yes Datadog-Client-Computed-Top-Level
DEFAULT 2025-11-13T10:09:57.581510Z 2025-11-13 10:09:57 UTC | SERVERLESS_INIT | DEBUG | Serializing 1 tracer payloads.
DEBUG 2025-11-13T10:09:57.781680Z Updated DatadogSampler with 5 service based sampling rates (provided by the agent)
DEFAULT 2025-11-13T10:09:57.781695Z Updated DatadogSampler with 5 service based sampling rates (provided by the agent)
DEFAULT 2025-11-13T10:09:57.781954Z 2025-11-13 10:09:57 UTC | SERVERLESS_INIT | DEBUG | Reported agent rates: target_tps=10 errors_tps=10 rare_sampling=false
DEFAULT 2025-11-13T10:09:57.782143Z 2025-11-13 10:09:57 UTC | SERVERLESS_INIT | DEBUG | Send messages for pipeline logs (msg_count:37, content_size=58407, avg_msg_size=1578.57)
DEFAULT 2025-11-13T10:09:58.180788Z 2025-11-13 10:09:58 UTC | SERVERLESS_INIT | DEBUG | Flushed stats to the API; time: 2.598442925s, bytes: 1113
DEFAULT 2025-11-13T10:09:58.381077Z 2025-11-13 10:09:58 UTC | SERVERLESS_INIT | DEBUG | Received a Flush trigger
DEFAULT 2025-11-13T10:09:58.381520Z 2025-11-13 10:09:58 UTC | SERVERLESS_INIT | DEBUG | Demultiplexer: sendIterableSeries: start sending iterable series to the serializer
DEFAULT 2025-11-13T10:09:58.381534Z 2025-11-13 10:09:58 UTC | SERVERLESS_INIT | DEBUG | The payload was not too big, returning the full payload
DEFAULT 2025-11-13T10:09:58.480721Z 2025-11-13 10:09:58 UTC | SERVERLESS_INIT | DEBUG | Flushed traces to the API; time: 300.049061ms, bytes: 1796
DEFAULT 2025-11-13T10:09:58.480737Z 2025-11-13 10:09:58 UTC | SERVERLESS_INIT | DEBUG | SyncForwarder has flushed 1 transactions
DEFAULT 2025-11-13T10:09:58.480743Z 2025-11-13 10:09:58 UTC | SERVERLESS_INIT | DEBUG | Demultiplexer: sendIterableSeries: stop routine
Libraries in Use
aiodns==3.2.0
aiofiles==23.2.1
aiohappyeyeballs==2.4.4
aiohttp==3.11.10
aioresponses==0.7.8
aiosignal==1.3.1
annotated-types==0.7.0
anyio==4.7.0
asyncpg==0.30.0
asyncpg-stubs==0.30.1
attrs==24.2.0
backoff==2.2.1
black==24.10.0
buildpg==0.4
bytecode==0.16.0
cachetools==5.5.0
certifi==2024.8.30
cffi==1.17.1
cfgv==3.4.0
chardet==4.0.0
charset-normalizer==3.4.0
click==8.1.7
coverage==7.6.9
cryptography==44.0.0
datadog==0.52.1
ddtrace==3.17.1
Deprecated==1.2.15
distlib==0.3.9
envier==0.6.1
fastapi==0.115.12
filelock==3.16.1
filetype==1.2.0
frozenlist==1.5.0
gcloud-aio-auth==5.3.2
gcloud-aio-storage==9.3.0
google-api-core==2.23.0
google-auth==2.36.0
google-cloud-pubsub==2.31.1
googleapis-common-protos==1.66.0
grpc-google-iam-v1==0.14.0
grpcio==1.68.1
grpcio-status==1.62.3
gunicorn==23.0.0
h11==0.14.0
hiredis==3.2.1
httpcore==1.0.7
httpx==0.28.1
identify==2.6.3
idna==3.10
importlib_metadata==8.5.0
iniconfig==2.0.0
libcst==1.5.1
mergedeep==1.3.4
moreorless==0.4.0
multidict==6.1.0
mypy-extensions==1.0.0
nodeenv==1.9.1
numpy==1.26.4
opencv-python-headless==4.11.0.86
opentelemetry-api==1.28.2
opentelemetry-sdk==1.28.2
opentelemetry-semantic-conventions==0.49b2
orjson==3.11.3
packaging==24.2
pathspec==0.12.1
pillow==11.0.0
platformdirs==4.3.6
pluggy==1.5.0
pre_commit==4.0.1
propcache==0.2.1
proto-plus==1.25.0
protobuf==4.25.5
pyasn1==0.6.1
pyasn1_modules==0.4.0
pycares==4.5.0
pycparser==2.22
pydantic==2.10.3
pydantic-settings==2.6.1
pydantic_core==2.27.1
Pygments==2.18.0
PyJWT==2.10.1
pyright==1.1.404
pytest==8.4.2
pytest-asyncio==0.23.8
pytest-cov==6.0.0
pytest-mock==3.14.0
pytest-subtests==0.12.1
pytest-watcher==0.4.3
python-dotenv==1.0.1
python-json-logger==2.0.7
python-multipart==0.0.20
PyYAML==6.0.2
redis==5.2.1
requests==2.32.3
rsa==4.9
ruff==0.3.7
setproctitle==1.3.7
sniffio==1.3.1
starlette==0.46.2
stdlibs==2024.12.3
toml==0.10.2
tomlkit==0.13.2
trailrunner==1.4.0
typing_extensions==4.15.0
ufmt==2.8.0
urllib3==2.2.3
usort==1.0.8.post1
uvicorn==0.32.1
uvicorn-worker==0.3.0
virtualenv==20.28.0
watchdog==6.0.0
wrapt==1.17.0
yarl==1.18.3
zipp==3.21.0
Operating System
GCP Cloud Run, unsure of host details