Handle non-serializable objects in vllm bench

[huydhn]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results

Purpose

#13993 added vllm bench cli but it has a small bug in how it fails to handle non-serializable objects when writing the benchmark results. The bug has been fixed in the nightly benchmark script in #19114, but vllm bench command didn't have the fix yet. This PR brings the same fix from #19114 over.

The issue https://github.com/pytorch/pytorch-integration-testing/actions/runs/16541559146/job/46783496249#step:14:3335 manifested in nightly benchmark run after #21355 started to switch to vllm bench in nightly benchmark without bringing over the fix in #19114. The code under benchmarks will be cleaned up in the future anyway, but I leave it to @yeqcharlotte to decide on the timing instead of cleaning them up here.

Test Plan

I'll post the benchmark run using the code from this PR in a min.

cc @yeqcharlotte @houseroad @simon-mo

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request adds a default handler to json.dump to handle non-serializable objects in benchmark results. A suggestion has been made to also handle NaN values by adding allow_nan=True to the json.dump call.

[yeqcharlotte]

Thanks so much for the fix! @huydhn anywhere else on top of your mind that we should double check?

[huydhn]

Thanks so much for the fix! @huydhn anywhere else on top of your mind that we should double check?

IMO, for changes w.r.t vllm bench, doing a quick round of testing with https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml is recommended. Take this PR as an example, after the Docker image was built and ready, I ran a quick benchmark with the following parameters:

• Branch set to refs/pull/21665/head
• Commit set to f8de642d90fa87e1dbf9e740e74d5f55c7a28870
• Model set to meta-llama/Meta-Llama-3.1-8B-Instruct as I only need one example
• Runner set to h100, same reason, just need an example

The benchmark signals from https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml have proved useful to catch some non-blocking issues like this one or #21476 (comment). So, I plan to setup a notification next when it starts failing.