Support specify template in OpenAI server #1263
Conversation
|
Well, it looks like we are working on somewhat related features. Not exactly the same thing, but these would be incompatible if both PRs were accepted right now. |
|
@Tostino Your implementation does offer more flexibility. However, I personally prefer using Prompt templating introduced in transformers 4.34.0. This would allow us to not rely on fastchat and to create custom templates stored in the tokenizer config files. Admittedly, this represents a significant change. |
|
Right. I agree something like that is a much better solution...but I am unable to use my model with vLLM right now without patching it, and that was the solution that was suggested to me in Discord, so I went with it and submitted a PR because it is useful "right now". I'd expect it to be replaced when an implementation of HF prompt templates is in place, and I very much look forward to this madness being solved. |
|
Just FYI, I am going to work on a PR for supporting the HF tokenizer prompt templating. Closed my other one since it hadn't been accepted yet. |
|
Very nice! I will close this PR since the HF tokenizer prompt templating work |
Currently, the OpenAI API server uses
request.modelas the template name. This approach isn't convenient when deploying custom models (e.g., a custom fine-tuned chat model that uses the chatgpt/llama2 template).This patch addresses the issue by introducing a
templatefield in theCompletionRequest/ChatCompletionRequest. If the template field isn't specified, it will default to the value ofmodelto maintain backward compatibility.