add benchmark for fix length input and output #5857
Conversation
|
To speed up the CI queue, I've cancelled the distributed tests for the latest CI run in this PR since they won't pass anyway until #5905 has been merged. Now that it has been merged, please merge |
I have merged from the main functiion |
There was a problem hiding this comment.
Hey @haichuan1221! Sorry for the late review and thank you for the contribution!
Overall LGTM and I have left a few comments/suggestions
benchmarks/benchmark_serving.py
Outdated
| return sampled_requests | ||
|
|
||
|
|
||
| def sample_random_requests(input_len, output_len, num_prompts, range_ratio, |
There was a problem hiding this comment.
Please add type hint for parameters in the function signature
There was a problem hiding this comment.
Just fixed, please check it again
There was a problem hiding this comment.
Update the formatting issues for you. For future reference, you can run the format.sh we provide in the repo for easily formatting your code changes.
There was a problem hiding this comment.
OK, I will. Thinks for the tip
benchmarks/benchmark_serving.py
Outdated
| parser.add_argument("--random-input-len", | ||
| type=int, | ||
| default=1024, | ||
| help="random sample input length") |
There was a problem hiding this comment.
| help="random sample input length") | |
| help="Number of randomly sampled input tokens per request, used only for random dataset") |
benchmarks/benchmark_serving.py
Outdated
| parser.add_argument("--random-output-len", | ||
| type=int, | ||
| default=128, | ||
| help="random sample output length") |
There was a problem hiding this comment.
Please update the help message per suggestion above.
benchmarks/benchmark_serving.py
Outdated
| parser.add_argument("--random-range-ratio", | ||
| type=float, | ||
| default=1.0, | ||
| help="random sample range ratio") |
There was a problem hiding this comment.
Please update the help message per suggestion above.
Co-authored-by: Roger Wang <[email protected]>
Co-authored-by: Roger Wang <[email protected]> Signed-off-by: Alvant <[email protected]>
Co-authored-by: Roger Wang <[email protected]> Signed-off-by: LeiWang1999 <[email protected]>
3 participants
vllm only support benchmark for sonnet and sharegpt dataset, but the input and output length is not fixed; in the contribution, I use random sample to evealuate the benchmark of fixed input length and output length