[NIXL][Misc] Expose metrics from NIXL for logging to CLI

[NickLucche]

This PR exposes the telemetry which is available in NIXL starting with v0.6.0 for logging to CLI.

Telemetry can be enabled in NIXL by setting:

NIXL_TELEMETRY_ENABLE=1/y/yes/on

Output example in vLLM:

(APIServer pid=205253) INFO 09-22 13:28:53 [metrics.py:98] KV Transfer metrics: Num successful transfers=1, Avg xfer time (ms)=9.862, P90 xfer time (ms)=9.862, Avg post time (ms)=5.693, P90 post time (ms)=5.693, Avg MB per transfer=70.0, Throughput (MB/s)=7097.952, Avg number of descriptors=56.0

My plan is to match this behavior for the current release, and then switch to a default "telemetry-on" mode starting with the next one (@markmc looking for feedback on this), so that a single on/off logging toggle can be maintained throughout vLLM.
The upcoming NIXL release will also allow setting telemetry through config (thanks @mkhazraee ) so I would propose we get rid of the env var handling altogether with the next upgrade.

Current metrics being tracked:

• num_successful_transfers
• transfer_duration
• post_duration
• bytes_transferred
• num_descriptors

I would appreciate feedback here on any other metrics you see fit (cc @robertgshaw2-redhat @tlrmchlsmth and other llm-d power-users) or any other derived ones (see reduce()) .

In particular regarding failures, I will sync with @njhill on upcoming PRs to handle per-request/per-block failures.
At this moment, from the nixl interface perspective, we just crash on recognized failed transfers

vllm/vllm/distributed/kv_transfer/kv_connector/v1/nixl_connector.py

Lines 1119 to 1124 in 3d2c56b

	elif xfer_state == "PROC":
	in_progress = True
	continue
	else:
	raise RuntimeError("Transfer failed with state %s",
	xfer_state)

.

Prometheus

The small set of current metrics exposed is for cli-only, but will follow-up with a separate PR to expose them to Prometheus.
Current design though should be very much compatible with Prometheus as is, with all stats being represented by an "Histogram" object (as per @markmc guidance).

Anything inside def reduce() are summary stats for cli that I do not expect to need in Prometheus. All other "raw" metrics collected should just be ingested and be available for custom aggregation by the logging engine.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @NickLucche.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[gemini-code-assist]

Code Review

This pull request introduces telemetry for NIXL, exposing various metrics for logging to the CLI. The changes include updating the nixl dependency, adding logic to collect transfer statistics when telemetry is enabled via an environment variable, and implementing aggregation and reduction of these stats for display. The implementation is well-structured, but I've identified a potential ZeroDivisionError in the metrics reduction logic that could cause a crash. My feedback includes a suggestion to prevent this.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @NickLucche.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[NickLucche]

Addressed changes here @markmc , thanks!

[markmc]

This string template thingy is repeated twice in the same function. Very minor nit