How to using stream output with vllm?

[lucasjinreal]

Although there are some lib wrappered vllm like TGI, but I want to know how to using vllm with stream output enabled, currently hard to found out-of-box example on it.

Typically, with original hf transforemrs API, one can using a TextStreamer and send into generate_kwargs to do this:

generate_kwargs = dict(
                **input_ids,
                max_new_tokens=50 if args.bare else 800,
                streamer=streamer,  <--- streamer
                do_sample=True,
                num_beams=1,
                temperature=float(args.temp),
                top_k=40,
                top_p=float(args.top_p),
          
            )

Any out-of-box I can use to enable stream in vllm? Thanks

[Huangyajuan-123]

i wanna learn this problem.

[lucasjinreal]

I solved it.

[FarziBuilder]

Please tell how did you solve it?

[lonePatient]

@lucasjinreal Please tell how did you solve it?

[Marten2016]

@lucasjinreal Please tell how did you solve it?

+1

[F5Psyche]

How did you solve it? Can you provide Python code reference?

[kevmo314]

This thread is hilarious, some true DenverCoder9 energy.

Anyways, there is some example code for how to do streaming output here: https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/api_server.py#L56-L63

[lucasjinreal]

The real hilarious thing is that I solved it and again after reviewing mess of vllm documentation, I come back to my own issue.

For thoese who using LLM wants streaming you can refer to:

results_generator = llm.generate(inputs, sampling_params, str(time.monotonic()))
    outputs = ""
    async for request_output in results_generator:
        prompt = request_output.prompt
        if request_output.finished:
            print()
        else:
            out = request_output.outputs[-1].text
            if len(out) == 0:
                continue
            out_delta = out[len(outputs) :]
            print(out_delta, end="", flush=True)
            outputs = out

One can get the output by concating the outputs, or yeild the output for API serving.

For now, I am stuck at the MLLM streaming issue, it's lenght not correct.

[wikithink]

just use
python -m vllm.entrypoints.openai.api_server --served-model-name