[Feature]: Add CLI Commands for Benchmarking

[simon-mo]

🚀 The feature, motivation and pitch

Currently the vllm commands has support for serve|chat|complete, while the benchmark commands lives in https://github.com/vllm-project/vllm/tree/main/benchmarks.

It would be great to have commands such as following

vllm benchmark-throughput
vllm benchmark-latency
vllm benchmark-serving

Alternatives

Status quo.

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[aarnphm]

wdyt about

vllm bench [throughput|latency|serving] --opts

vllm bench --all

[simon-mo]

vllm bench [throughput|latency|serving] --opts works too. The main problem is that latency and throughput host LLMEngine within the process while serving is simply a load testing client (require external server)

[jeejeelee]

Are we considering tuning the kernel, such as something similar to vllm kernel-tuning --name moe?

[aarnphm]

LLMEngine within the process while serving is simply a load testing client (require external server)

I see, we also have benchmark_serving_guided.py and benchmark_guided.py for online and offline use-case respectively. fwiw I think bench <opts> can be running benchmark on server, then maybe we can add a --offline for a offline benchmark

[randyjhc]

Hi! I'd like to work on this.

[plops655]

I did benchmark-throughput and benchmark-latency and will merge my changes. Can someone do benchmark-serving. As aarnphm mentioned, there are several cases to deal with depending on whether we are online or offline serving.

[plops655]

Does anyone have any idea what issue I am running into. I added a benchmark.py file into the vllm/entrypoints/cli and set

def cmd_init()->List[CLISubcommand]:
return [BenchmarkThroughputCommand(), BenchmarkLatencyCommand()]

I then added vllm.entrypoints.cli.benchmark to the CMD_MODULES in vllm/entrypoints/cli/main.py but vllm benchmark-throughput is not recognized.

I plan on debugging this tomorrow.