[Feature] Add vllm bench CLI

[randyjhc]

Part of #13840

This PR adds CLI commands of vllm bench. We only support vllm bench serve in this PR to align the interface, the following benchmark modes will be added in follow-up PRs (contributions are welcome).

vllm bench latency
vllm bench throughput

What has been covered in this PR:

• All metrics in vllm bench serve.
• OpenAI endpoint request function.
• Random dataset

Future work:

• Serve
  • Support other request functions (e.g., TGI).
  • Support other datasets (e.g., ShareGPT, sonnet, etc).
• Latency
  • Support vllm bench latency.
• Throughput
  • Support vllm bench throughput.
• Refactor benchmarks/ to use this CLI.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[comaniac]

Thanks for the PR! My two cents:

1. We could still use vllm benchmark [throughput|latency|serving] with hierarchal CLIs. For example, it shows different arguments/options with vllm benchmark serve --help and vllm benchmark latency --help. If the current CLISubcommand cannot support this we should consider improving it.
2. I agree with that. We could move all common utilities under vllm/benchmark. Ideally once this CLI is landed, the current benchmarks folder should just have some shell scripts that use the CLI.
3. We should not put datasets to the package. Instead, we could upload them to the vLLM S3 buckets and put the link to the default value of the dataset argument.

cc @ywang96 @simon-mo

[randyjhc]

Thanks for the suggestions! I’ll keep working on improving points 1 and 3.

[randyjhc]

The new commits introduce the nested version for benchmark CLI.

• To invoke the benchmark:

vllm bench [throughput|latency|serving] --opts

• To view help messages:

vllm bench --help
vllm bench [throughput|latency|serving] --help

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @randyjhc.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[comaniac]

@khluu do you know why the changes made by local pre-commit is different from the CI and how should we fix it?

[khluu]

@khluu do you know why the changes made by local pre-commit is different from the CI and how should we fix it?

A bug fix related to precommit was merged not too long ago.. maybe merge this branch with main and try again?

[randyjhc]

Reframing the PR

1. Currently, this PR only supports vllm bench serving --opts at this time, but it can be easily extended to support other benchmark types like:

vllm bench [bench_type] --opts

2. At this stage, the vllm bench serving command only supports random requests, leaving sonnet.txt in place. In addition, the supported backends are limited to:

vllm
lmdeploy
openai
scalellm
sglang

3. Instead of moving the entire benchmarks folder, I have copied only the necessary functions into vllm/benchmarks. This approach helps us merge new changes in benchmarks in subsequent PRs while preventing conflicts for now.

Follow-up Items

1. Extending support for additional benchmark command types within this framework, such as:

vllm bench [latency|throughput]

2. Enabling full support for options in vllm bench serving --opts, including:

• Existing datasets such as sonnet, sharegpt, etc.
• Backend options, like tgi, mii, etc.

[comaniac]

It's in a good shape in general. Will take a deep look tomorrow.
Meanwhile please update the PR description with the latest scope of this PR and the remaining follow-up tasks.

[ywang96]

Hey @randyjhc! Thanks for working on this PR!

Now that #14036 has landed, could you include the dataset file into vllm/benchmarks too and update this PR accordingly?

[comaniac]

Hey @randyjhc! Thanks for working on this PR!

Now that #14036 has landed, could you include the dataset file into vllm/benchmarks too and update this PR accordingly?

We have to options to move forward:

1. We only support random sampling in this PR to keep it short, and focus on the interface/CLI.
2. We support datasets in this PR too to deliver a fully function benchmark serving capability.

I actually prefer (1) because it should take less time to merge this PR, but if you feel supporting datasets won't postponing the review process (because the logic is mostly copied), I'm also ok with it.

[ywang96]

Hey @randyjhc! Thanks for working on this PR!
Now that #14036 has landed, could you include the dataset file into vllm/benchmarks too and update this PR accordingly?

We have to options to move forward:

1. We only support random sampling in this PR to keep it short, and focus on the interface/CLI.
2. We support datasets in this PR too to deliver a fully function benchmark serving capability.

I actually prefer (1) because it should take less time to merge this PR, but if you feel supporting datasets won't postponing the review process (because the logic is mostly copied), I'm also ok with it.

yea I'm also okay with (1) just to get this PR in!

[ywang96]

LGTM!