[Feature]: Add OpenTelemetry (trace) API to v1

[JeffLuoo]

🚀 The feature, motivation and pitch

#4687 adds the OpenTelemetry support to request handling in vLLM. However, this feature is not enabled if VLLM_USE_V1=1 is configured.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[chaunceyjiang]

@DarkLight1337 @hmellor WDYT?

[DarkLight1337]

cc @njhill @markmc

[markmc]

See the V1 metrics design doc for my take:

Metrics provide an aggregated view over time of the system’s performance and health. Tracing, on the other hand, tracks individual requests as they move through different services and components. Both fall under the more general heading of “Observability”.

Since metrics is a big enough topic on its own, we are going to tackle the topic of tracing in v1 separately.

In other words, I personally haven't formed an opinion on it 🤷‍♂ I'd like to hear more about the use case, because it doesn't appear to be have been a blocker for V1 adoption to date.

[JeffLuoo]

This is not about a specific use case but a general tracing support in V1 which provides insights into individual request (or failing request based on the sampling decision), and for a request, there might be other components before vLLM with tracing support so having vLLM support tracing, we can chain the request together in one view. I also don't think it's going to be a blocker but a nice to have feature to V1 given V1 is the main version to use with vLLM at the moment.

[markmc]

general tracing [..] provides insights into individual request (or failing request based on the sampling decision), and [..] chain the request together in one view [of all components involved].

Right, that's what OpenTelemetry support provides, but the "use case" I'd like to hear more about is the why? In what context is this useful? Hearing that answer directly from users is super useful in helping to justify the feature, but also to understand what granularity of tracing is actually needed

[JeffLuoo]

Tracing could provide following:

• Troubleshoot individual request with long latency or failing request.
• For the non-deterministic nature of GenAI agents with multiple components, the trace could help to figure out which step in the services the request fails (together with metadata provided in each step following the OTLP GenAI SemConv). It provides insights into decision-making processes.
• Correlation with logs if logs contain trace/span ID in the entry, so logs/traces will be used together to provide more information for troubleshooting like: "What's the prompt/response for request with long latency?", "When the request fails, does it crash the vLLM with stacktraces in logs?"

[RichardoMrMu]

I try engine v1 tracing these days, is this support for your need? #17874

[xinli-sw]

Thanks for this initiative! I’ve been working on something related and would love to contribute / discuss

Hey @markmc , I was browsing the issue list and found this #16473 to be relavant. Though I'm not sure if this issue is widely observed by others. However, having otlp support is generally good for these kind of things.,

Right, that's what OpenTelemetry support provides, but the "use case" I'd like to hear more about is the why? In what context is this useful? Hearing that answer directly from users is super useful in helping to justify the feature, but also to understand what granularity of tracing is actually needed

If we could enable tracing support outside of main model execution this would be already great (understanding the lifecycle of a request through engine core, preprocessors, post-processing would be great).

Speaking from performance engineering perspective, GPU activity is easier to reason about (over the years of battling with them) and trace through using nsys is easy, but increasingly we found performance bottleneck to be inter-related to scheduling decisions and other parts of the model,

[tomasruizt]

OTEL support would be appreciated! 🚀 As a user, I'm facing problems with timeouts, which is surprising to me since the aggregate vLLM metrics seem healthy.

My scenario: I'm running a small model (Qwen/Qwen3-4B) on a large GPU (H100 96GB), and set a request timeout of 10min. This is a very long time imo. I send 1000 requests with client-side concurrency=64 (aiohttp). The vLLM aggregate metrics don't show obvious problems (0 waiting requests all the time, GPU KV cache Usage around 30%). Therefore, the aggregate metrics appear insufficient to diagnose the problem (or perhaps I'm just missing something). I attach the logs and metrics plot for reference: logs.tx, Metrics Plot

Using a request-level trace I could investigate how the failing requests differ from requests that succeed, and if the problem lies somewhere in their scheduling.