Server concurrency, streaming, and ssl

[tim-eastwood]

It would be amazing if the llama.cpp server had some features to make it suitable for more than a single user in a test environment

eg.:

• use a non-blocking server
• SSL support
• streamed responses

As an aside, it's difficult to actually confirm, but it seems like the n_keep option when set to 0 still actually keeps tokens from the previous prompt. Also, the help text lists --keep as a cmd line switch but it's not recognized when used (I wonder if that's part of the issue)

Thanks for reading

[SlyEcho]

The server has been updated now, but it does not solve all your issues. It can pretty much handle only one user/application effectively.

Streaming responses have been added.

SSL will probably never be added, you can use some kind of reverse proxy if you really need it.

[tim-eastwood]

I know this is probably not a valid comparison, but using HF transformers and pytorch in python, I was able to achieve this by globally loading the model at runtime, and calling the inference on it from within the server request handler which by nature of the library is already thread managed and asynchronous. that way the model doesn't need to be loaded for every client, and it also allows the client to choose their own settings like temperature, top_k, and so on, as those belong to the inference rather than the model.

I just mention this as I wonder if implementing an async server library would offer the same benefit of inherently solving the threading / sync issue without having to do anything to the way inference works.

If your response is "you have no idea how you C++ works", you would be totally correct XD Worth a try though!

[SlyEcho]

It should be possible, but right now the inference state is a little too global. But the state itself is very big (could be over a gigabyte) if you wanted to create one for every incoming connection you could easily run out of system or GPU memory. But then it could serve multiple connected clients. It doesn't even need to be concurrent, just handling I/O events whenever needed or with some kind of queue. This is actually normal for servers, you can get very high performance even without using multiple threads.

[ggerganov]

#1797 adds support for parallel contexts with a single model load.

Technically, when concurrent generations are running, one can naively switch the contexts for every token for every generation. But this is far from optimal, because the speed for a client would degrade proportionally to the concurrent generations at a given time.

Batching the generations as @SlyEcho described would be much more efficient, but I'm still not sure what would it take to demonstrate this approach. It might be quite difficult or it might be super easy -- haven't thought deeply about it yet. Would be definitely very valuable to have some implementation of that, since it is a common use case when serving models

[SlyEcho]

Batching would be more efficient, yes, but it's also hugely complex, I'm just thinking about how to coordinate multiple clients generating at the same time while keeping acceptable latency.

[tim-eastwood]

I for one am in favour of the "switching context for each token" approach. Leaving concerns about the server's capacity up to the admin puts the onus on them to monitor and manage the concurrent workload. Not to mention with the right resources (side scaling of server instances and a load balancer) this is less of a limitation as opposed to being the only real way to deploy this at any sort of scale.

I'm inferencing 7b models at around 30 tokens/s (not with ggml), which is more than capable of inferencing to about 5 or 6 users before the speed is too degraded. If I had to queue requests I would need to implement a way to maintain a connection for seconds? (minutes even?) until the front of the queue is reached, and also add to the front-end a way to communicate the queue status to the user, etc etc.

Either method is feasible but especially if parallel contexts already exists it seems to make the most sense. I'll add that in my case I have pretty overpowered servers for the models I'm hosting, but I respect that other people will want to host this on heavily limited hardware

Cheers @ggerganov

[okpatil4u]

Would vllm be helpful here by any chance ? There is an issue for it #1955

…

On Sat, 24 Jun 2023 at 11:36 PM, Henri Vasserman ***@***.***> wrote: Batching would be more efficient, yes, but it's also hugely complex, I'm just thinking about how to coordinate multiple clients generating at the same time while keeping acceptable latency. — Reply to this email directly, view it on GitHub <#1871 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXGU4AQIFAQGHZZQ34KUMTXM4UCFANCNFSM6AAAAAAZHVEEUA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[PenutChen]

This is insane, thanks for the recommendation!

[ianscrivener]

Non C++ dev here
when I read this thread the words of a wise man flashed into my mind
"Bloating the software with the ideas of today will make it useless tomorrow."

If there's a chance of a performance downside - IMO I'd rather a blazingly fast 'single threaded' app than a slow 'multi-threaded' one.

"Inference at the edge" hints at 2 things:
(1) not a server
(2) lower spec hardware

What about ggml super performant save/resume session functionality (ie memory & context). This would be great functonality for even the lowest spec hardware (say Raspberry Pi). Higher spec hardware could have n sessions saved into RAM instead of 'disk'.
--max-sessions 16
--max-session-memory-gb 8