how to use chat function

[zhangzai666]

I found that vllm could only use the generate method. like llm.generate(prompts, sampling_params)..I now want to use the chat method of qwen-chat-7b like llm.chat(prompts, sampling_params)

[Palmik]

Easiest would be for you to start the vllm server, which offer OpenAI compatible API.
Here: https://vllm.readthedocs.io/en/latest/getting_started/quickstart.html#openai-compatible-server

[zruiii]

Try to use llm.generate(prompt_token_ids, sampling_params) instead. Pass the tokens into generate function, but not text.

[aemonge]

Hmmm, but that would not create a chat.

Is there a way we can use this to create a chat like prompts, without creating a server?

r = llm.generate("Hi, my name in Andres")
print(r)  # "Hello Andres"

[hammoudhasan]

@aemonge did you ever figure this out ? What I'm currently doing is using the apply_chat_template from the huggingface tokenizer and then passing it through. Probably not ideal !

[aemonge]

I think I did, but I can't recall it right now.

What I can do is share my project with you, which is a Chat CLI that watches I/O files as input/output.

Would you like to see it in my project?