[Feature]: Ensure benchmark serving do not import vLLM

[simon-mo]

🚀 The feature, motivation and pitch

vLLM's benchmark serving script is expected to be a standalone inference client that only requires minimum dependencies. Currently, it still imports vllm conditionally.

The task is as follows:

1. Clearly define a requirements txt for benchmark serving client

numpy
pandas
Pillow
tqdm
transformers
datasets

2. Add a CI test that create a new uv environment and execute the script. Ensure there is no vLLM present. This can be part of existing tests for benchmark scripts. https://github.com/vllm-project/vllm/blob/main/.buildkite/run-benchmarks.sh

3. Make sure the existing usage of vLLM is moved to inlining whatever utility method is required.

Alternatives

No response

Additional context

See #14879 for discussion, cc @houseroad @ywang96

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Kelvinyu1117]

I would like to help with this task.
I presume you are referring to benchmarks/benchmark_serving.py right?
For the requirements.txt, should it apply to all the scripts in the benchmark directory?

[russellb]

This seems odd if there's also work to move the benchmark scripts to be within vllm as a subcommand.

#13993

I think ensuring vllm bench remains functional with a minimal dependency set makes sense, but it'll be impossible to have that without installing vllm itself.

[houseroad]

I think they are conflicting with each others, we should just decouple the vllm deps from benchmark client. And the CLI can just reuse it.

[houseroad]

cc: @yangw-dev

[ywang96]

My take re: benchmark CLI vs benchmark scripts after discussing with @comaniac

1. The benchmark serving script (and perhaps some other scripts too) under benchmarks should be portable so that any user and developers can easily git clone and run them without any heavy dependencies (i.e needing to install vllm the package)
2. The benchmark CLI is more tailored to vLLM users/developers as a quick shortcut to verify the performance of vLLM inference server (so instead of having to run python3 benchmarks/benchmark_throughput.py ......., one may simply do something like vllm bench offline), and for online inference, vllm bench serve can also include the process to set up the inference server with some default recommended configurations

[russellb]

Having both seems like a bad idea to me ... I'd much rather we pick one or the other and focus all energy on that.

[ywang96]

@simon-mo Maybe you can chime in here? since you made the original issue to add a CLI functionality for benchmarks.

[aarnphm]

Agreed on Russell's point here, I think we should either do CLI or having the benchmarks directory.

I think having CLI or having the benchmark repo is equivalent

git clone https://github.com/vllm-project/vllm && cd vllm/benchmarks && ... versus uv run vllm bench ...

[Kelvinyu1117]

I think having a benchmark as a subcommand doesn't add much benefit to the user, given that the user can run the script directly.

[russellb]

I think having a benchmark as a subcommand doesn't add much benefit to the user, given that the user can run the script directly.

It's a bit nicer of a UX (just an opinion).

but it also allows running it via standard installation mechanisms (install vllm) instead of having to clone the git repo and run from there.

If the concerns on each side are UX vs limiting dependencies, we could accomplish both goals here. It doesn't have to be vllm bench. We could make it its own independent Python package that installs a vllm-bench command, for example.