Why llama-server is so limited to concurrent requests even using -cb and -np ?

[celsowm]

Hi
Why llama-server is so limited to concurrent requests even using -cb and -np ?
Sglang and vllm servers are not so limited like llama.cpp when you have even hundreds of requests

o3 response:

vLLM and SGLang were designed from the ground-up for token-level scheduling and smart KV-cache management, so one copy of the model can keep a GPU busy while dozens of users stream tokens concurrently.
llama-cpp’s server, by contrast, simply round-robins request-level contexts that each own a large, contiguous cache, so every extra user slices throughput and memory almost linearly.