[Feature] Add support to concurrently calculate prefills of multiple requests

[kelliaao]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

We have a problem where high volumes of small-prompt requests are usually processed smoothly, but quickly pile up in a giant queue when a small number of large-prompt requests are submitted.We attempted to use the --enable-mixed-chunk parameter to enable sglang to handle both prefill and decode concurrently. However, sglang only processes a single sequence during prefill, causing short-text requests to remain blocked by long-text requests.
We also observed that vLLM allows concurrent processing of multiple requests in the prefill phase through the --max-num-partial-prefills parameter. Nonetheless, sglang demonstrably outperforms vLLM on DeepSeek series models, making us reluctant to abandon sglang. We hope sglang can also support this feature.

Related resources

vllm-project/vllm#10235

[Swipe4057]

try changing the scheduler from lpm to fcfs this should ease the blocking situation

[kelliaao]

@Swipe4057 I believe we are using this strategy, but when users initiate long texts, there are still occasional timeouts for short-text responses (no response for over 5 minutes), which is unacceptable in some applications.

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.