Support chat template and echo for chat API

[Tostino]

This pull request introduces the chat template feature to vLLM, utilizing the template stored in the tokenizer, enhancing its compatibility with the OpenAI Chat API.
https://huggingface.co/blog/chat-templates

This only affects the OpenAI API chat/completion endpoint, the regular completion endpoint does not utilize this feature.

There has already been a ton of discussion under this PR: #1493 but I accidentally messed things up by replacing the branch, so we are trying this again...

1. Addition of the --chat-template command-line argument to specify a chat template file or single-line template for the model.
2. Implementation of the --response-role command-line argument for defining the role name in chat responses when add_generation_prompt is set to true.
3. Update to the chat API request handling to support handling finishing a partial response correctly, and echoing input portions of messages (request.add_generation_prompt, and request.echo).
4. Addition of new chat templates examples (template_chatml.jinja, template_alpaca.jinja, and template_inkbot.jinja) showing the multiple ways they can be specified.
5. More robust error handling, and fix the responses to actually match the OpenAI API format.
6. Update quickstart.rst to show the new features.

[aarnphm]

@simon-mo whats your opinion on just supporting chat templates through envvar? By doing this, there would be no --prompt-template or additional file logic in argparse. We can redirect users to the HF chat templates docs on this.

vllm should be able to document how people can pass in the jinja templates through a envvar, so that vllm won't handle any parsing, and users are responsible for this?

I think this is powerful in a way that users have full control of the chat templates. By default, if there is none provided then fall back to huggingface behaviour?

I think we can provide a default chat templates if needed

edit: This is probably good for serverless as well.

[casper-hansen]

Can we also make sure there is a way to disable chat templates such that users provide messages that are preformatted?

[Tostino]

Can we also make sure there is a way to disable chat templates such that users provide messages that are preformatted?

As said before, you use the regular completion endpoint if you want to provide a pre-formatted string to the model.

Explain how we would provide the preformatted text using the chat/completion endpoint which requires a list of messages.
Here is an example call:

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer " \
  -d '{
    "model": "Tostino/Inkbot-13B-8k-0.2",
    "stop": ["\n<#user#>", "\n<#bot#>"],
    "stream": false,
    "add_generation_prompt": true,
    "echo": true,
    "n": 1,
    "messages": [
	{"role": "meta-current_date", "content": "2023-10-20"},
	{"role": "meta-task_name", "content": "kg"},
	{"role": "system", "content": "You are a helpful assistant."},
	{"role": "user", "content": "Hello!"},
	{"role": "assistant", "content": "Hello, how are you?"},
	{"role": "user", "content": "Great, thank you! Now i would like some help planning my weekend in Asheville, I would like some hike suggestions."},
	{"role": "assistant", "content": "Asheville is a fantastic destination for hiking enthusiasts! Here are a few suggestions:\n\n1. Black Balsam Knob Trail: This 4.4-mile round trip trail offers stunning views of the surrounding mountains.\n2. Max Patch Loop: This 2.4-mile loop is perfect for families or beginners, featuring wildflowers and mountain views.\n3. Shut-ins Trail: The 2.8-mile hike takes you through a scenic gorge with cascading waterfalls.\n4. Graveyard Fields: For a longer hike, this 8-mile loop takes you through meadows, streams, and a beautiful forest.\n5. Linville Gorge: If you are up for a more challenging hike, the Linville Gorge has various trails with breathtaking views."},
	{"role": "user", "content": "Extract key concepts and relationships from the conversation to form a knowledge graph."}	
	]
  }'

[casper-hansen]

I see where you are coming from - perhaps it should be mentioned in the PR that this only applies to certain endpoints.

[aarnphm]

Can we also make sure there is a way to disable chat templates such that users provide messages that are preformatted?

As said before, you use the regular completion endpoint if you want to provide a pre-formatted string to the model.

Explain how we would provide the preformatted text using the chat/completion endpoint which requires a list of messages. Here is an example call:

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer " \
  -d '{
    "model": "Tostino/Inkbot-13B-8k-0.2",
    "stop": ["\n<#user#>", "\n<#bot#>"],
    "stream": false,
    "add_generation_prompt": true,
    "echo": true,
    "n": 1,
    "messages": [
	{"role": "meta-current_date", "content": "2023-10-20"},
	{"role": "meta-task_name", "content": "kg"},
	{"role": "system", "content": "You are a helpful assistant."},
	{"role": "user", "content": "Hello!"},
	{"role": "assistant", "content": "Hello, how are you?"},
	{"role": "user", "content": "Great, thank you! Now i would like some help planning my weekend in Asheville, I would like some hike suggestions."},
	{"role": "assistant", "content": "Asheville is a fantastic destination for hiking enthusiasts! Here are a few suggestions:\n\n1. Black Balsam Knob Trail: This 4.4-mile round trip trail offers stunning views of the surrounding mountains.\n2. Max Patch Loop: This 2.4-mile loop is perfect for families or beginners, featuring wildflowers and mountain views.\n3. Shut-ins Trail: The 2.8-mile hike takes you through a scenic gorge with cascading waterfalls.\n4. Graveyard Fields: For a longer hike, this 8-mile loop takes you through meadows, streams, and a beautiful forest.\n5. Linville Gorge: If you are up for a more challenging hike, the Linville Gorge has various trails with breathtaking views."},
	{"role": "user", "content": "Extract key concepts and relationships from the conversation to form a knowledge graph."}	
	]
  }'

I think chat templates should only apply for /v1/chat/completions. Users can change the chat templates if needed. /v1/completions should still stay the way they were before, which means that users should have full responsibility for formatting their prompt.

[intrafindBreno]

This pull request is a great contribution to the vllm project. I hope it gets merged soon!

[dongxiaolong]

Hi @Tostino ,

Thank you so much for your PR! Your contribution has successfully addressed key issues in model inference, enabling the implementation of function calls using models like OpenHermes. Here is an example I created that demonstrates the effective application of OpenHermes: OpenHermes Functions with VLLM. Your work has truly unlocked the potential of OpenHermes. Adding another practical example to your contribution would greatly help others understand the significant impact of this work.

[tjtanaa]

Huggingface now supports chat_template through their tokenizer class. I think if the community are using this feature, it will make chat template tracking easier.

To that end, we have setup a huggingface space that allows developers and the community to encourage the use of huggingface tokenizer's chat_template. We have provided many of the well-known prompt templates. There are many more templates to come within these few weeks.

The ui interface also allows user to download chat template as jinja2 which I think would be beneficial for this PR feature as well.

[Tostino]

@tjtanaa That is a neat tool, glad there is more work going into this in the same direction. Being able to download the template to a file easily would be beneficial for users.
One thing to note, is that it incorrectly parses my template for Tostino/Inkbot-13B-8k-0.2 when I tried. The newlines are ignored.

@dongxiaolong Glad you found it useful! Very cool example to see working.

[tjtanaa]

@tjtanaa That is a neat tool, glad there is more work going into this in the same direction. Being able to download the template to a file easily would be beneficial for users. One thing to note, is that it incorrectly parses my template for Tostino/Inkbot-13B-8k-0.2 when I tried. The newlines are ignored.

@dongxiaolong Glad you found it useful! Very cool example to see working.

Thank you for taking the time to try it out.
I am following the huggingface example which stores the chat-template into a one-liner jinja2 code. So if you want to have a newline, you should try to add in + '\n' explicitly. e.g.

airoboros_v2.jinja2

{% if not add_generation_prompt is defined %}
{% set add_generation_prompt = false %}
{% endif %}
{% for message in messages %}
{% if not loop.first %}
{% endif %}
{% if message['role'] == 'system' %}
{{ message['content'] + '\n' }}
{% elif message['role'] == 'user' %}
{{ 'USER: ' + message['content'] + '\n' }}
{% elif message['role'] == 'assistant' %}
{{ 'ASSISTANT: ' + message['content'] + '</s>' }}
{% endif %}
{% endfor %}
{% if add_generation_prompt %}
{{ 'ASSISTANT:' }}
{% endif %}

[Tostino]

@tjtanaa where can we discuss that issue other than on this PR?

My template is valid jinja, and it does have new lines embedded in the single-line version of it. There was an extra step to correctly load the single line jinja in this PR to deal with new lines properly, I'm guessing you just need to do the same.

Edit: I'll open a discussion on the HF repo.

[Tostino]

@aarnphm / @simon-mo / @WoosukKwon (or anyone else appropriate)

Is there anything that needs to be addressed prior to merging this PR at this point? There is nothing else I am aware of, as it's been pretty thoroughly tested.

[aarnphm]

I think this largely depends on the templates itself right? By default I don't think this is needed.

[Tostino]

The response-role? Some models may not use user/assistant as the role names. The templates are free for the creator to choose with that. I defaulted to the same behavior as the OpenAI API, but made it so there is compatibility with the flexibility provided by the chat_template feature.

[aarnphm]

Ye but we already have that in the message['role'] right (usually alternate between user and assistant, so for example in the phi-1.5 cases it would be BOB and SANDRA?) not sure why we need this

[Tostino]

Yes it usually alternates between user/assistant...but given the following request, how would we know the role to respond with?

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer " \
  -d '{
    "model": "Tostino/Inkbot-13B-8k-0.2",
    "stream": false,
    "n": 1,
    "messages": [
	{"role": "meta-current_date", "content": "2023-10-20"},
	{"role": "meta-task_name", "content": "general"},
	{"role": "system", "content": "You are a helpful assistant."},
	{"role": "user", "content": "What is the capital of the USA?"}	
	]
  }'

Previously, it was hard-coded as assistant. Now it is configurable with that cli argument. Ideally, this would be additional metadata about how to use the template that would be stored somewhere in the tokenizer...but HF didn't think that far ahead.

[aarnphm]

Users should be responsible for doing few-shot prompt right?

[Tostino]

No, that is totally unnecessary for a lot of models and just eats up context space. Not to mention that locks us into two-role conversations. What if there is a user_context role that is used for any input from a file the user wants to interact with, and that is appended after their text input. That is a real use case I have been using this implementation for.

[aarnphm]

Like here should it just be request.messages[-1]['role']?

[Tostino]

No, that doesn't work.

[Yard1]

@Tostino Would it be possible to add some simple unit tests for this?

[Tostino]

@Yard1 Sure, are there any existing tests for the server that I can add to?

[Yard1]

I don't think there are any for the OpenAI server specifically.

[Tostino]

@Yard1 So, to properly test this, it looks like I need to refactor a whole lot more code... (I could be mistaken...my dayjob is not python so i've never used any of the testing libraries before, and am learning on-the-fly)

Are you sure you want me to do that? I am not doing any more work that will be thrown away...I've spent far too much time on this already.

And it now looks like i'll have to spend more time rebasing, because there are conflicts again.

[simon-mo]

Thank you for adding the tests.

I made a pass for small code quality nits. Attached diff in this comment. @Tostino please let me know when you have finished adding tests so I can directly push to this branch without causing you more merge conflict and resolving them one by one.

I think for the future work, the design considerations still need to flush out with actual use cases. I'm aware of previous discussion on this, this PR should not be blocked by any of these. Just putting these here as notes:

• add_generation_prompt: is it really a request parameter or bound by the model itself. I have the feeling that most use case will always set it to one value and doesn't change across requests.
• echo for chat completion API: currently this PR prepend the previous role is the same (essentially a "continue" type of workflow during chat); because openai doesn't have the echo semantics defined for chat, we can think a bit more about exactly what's the semantics; and be aware of future changes.

Diff Attached

diff --git a/docs/source/getting_started/quickstart.rst b/docs/source/getting_started/quickstart.rst
index 1516a7b..bd3940f 100644
--- a/docs/source/getting_started/quickstart.rst
+++ b/docs/source/getting_started/quickstart.rst
@@ -124,11 +124,12 @@ Use model from www.modelscope.cn
     $     --model="qwen/Qwen-7B-Chat" --revision="v1.1.8" --trust-remote-code
 
 By default, the server uses a predefined chat template stored in the tokenizer. You can override this template by using the ``--chat-template`` argument:
+
 .. code-block:: console
 
    $ python -m vllm.entrypoints.openai.api_server \
-       --model facebook/opt-125m \
-       --chat-template ./examples/template_chatml.json
+   $     --model facebook/opt-125m \
+   $     --chat-template ./examples/template_chatml.json
 
 This server can be queried in the same format as OpenAI API. For example, list the models:
 
@@ -137,7 +138,7 @@ This server can be queried in the same format as OpenAI API. For example, list t
     $ curl http://localhost:8000/v1/models
 
 Using OpenAI Completions API with vLLM
---------------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Query the model with input prompts:
 
@@ -167,7 +168,7 @@ Since this server is compatible with OpenAI API, you can use it as a drop-in rep
 For a more detailed client example, refer to `examples/openai_completion_client.py <https://github.com/vllm-project/vllm/blob/main/examples/openai_completion_client.py>`_.
 
 Using OpenAI Chat API with vLLM
--------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 The vLLM server is designed to support the OpenAI Chat API, allowing you to engage in dynamic conversations with the model. The chat interface is a more interactive way to communicate with the model, allowing back-and-forth exchanges that can be stored in the chat history. This is useful for tasks that require context or more detailed explanations.
 
diff --git a/vllm/entrypoints/openai/api_server.py b/vllm/entrypoints/openai/api_server.py
index b87fcd9..54ff3c8 100644
--- a/vllm/entrypoints/openai/api_server.py
+++ b/vllm/entrypoints/openai/api_server.py
@@ -47,15 +47,22 @@ def create_error_response(status_code: HTTPStatus,
                         status_code=status_code.value)
 
 
-def load_chat_template():
-    try:
-        with open(args.chat_template, "r") as f:
-            content = f.read()
-            return content
-    except OSError:
-        # If opening a file fails, set chat template to be args to
-        # ensure we decode so our escape are interpreted correctly
-        return codecs.decode(args.chat_template, "unicode_escape")
+def load_chat_template(args, tokenizer):
+    if args.chat_template is not None:
+        try:
+            with open(args.chat_template, "r") as f:
+                chat_template = f.read()
+        except OSError:
+            # If opening a file fails, set chat template to be args to
+            # ensure we decode so our escape are interpreted correctly
+            chat_template = codecs.decode(args.chat_template, "unicode_escape")
+
+        tokenizer.chat_template = chat_template
+        logger.info(f"Using supplied chat template:\n{tokenizer.chat_template}")
+    elif tokenizer.chat_template is not None:
+        logger.info(f"Using default chat template:\n{tokenizer.chat_template}")
+    else:
+        logger.warning("No chat template provided. Chat API will not work.")
 
 
 @app.exception_handler(RequestValidationError)
@@ -73,16 +80,6 @@ async def check_model(request) -> Optional[JSONResponse]:
     return ret
 
 
-async def get_gen_prompt(request) -> str:
-    try:
-        return tokenizer.apply_chat_template(
-            conversation=request.messages,
-            tokenize=False,
-            add_generation_prompt=request.add_generation_prompt)
-    except Exception as e:
-        raise RuntimeError(f"Error generating prompt: {str(e)}") from e
-
-
 async def check_length(
     request: Union[ChatCompletionRequest, CompletionRequest],
     prompt: Optional[str] = None,
@@ -174,8 +171,6 @@ async def create_chat_completion(request: ChatCompletionRequest,
         - function_call (Users should implement this by themselves)
         - logit_bias (to be supported by vLLM engine)
     """
-    logger.info(f"Received chat completion request: {request}")
-
     error_check_ret = await check_model(request)
     if error_check_ret is not None:
         return error_check_ret
@@ -186,9 +181,12 @@ async def create_chat_completion(request: ChatCompletionRequest,
                                      "logit_bias is not currently supported")
 
     try:
-        prompt = await get_gen_prompt(request)
-    except RuntimeError as e:
-        logger.error(f"Error in generating prompt from request: {str(e)}")
+        prompt = tokenizer.apply_chat_template(
+            conversation=request.messages,
+            tokenize=False,
+            add_generation_prompt=request.add_generation_prompt)
+    except Exception as e:
+        logger.error(f"Error in applying chat template from request: {str(e)}")
         return create_error_response(HTTPStatus.BAD_REQUEST, str(e))
 
     token_ids, error_check_ret = await check_length(request, prompt=prompt)
@@ -198,7 +196,7 @@ async def create_chat_completion(request: ChatCompletionRequest,
     model_name = request.model
     request_id = f"cmpl-{random_uuid()}"
     created_time = int(time.monotonic())
-    obj_str = "chat.completion.chunk"
+    chunk_object_type = "chat.completion.chunk"
     try:
         spaces_between_special_tokens = request.spaces_between_special_tokens
         sampling_params = SamplingParams(
@@ -236,7 +234,7 @@ async def create_chat_completion(request: ChatCompletionRequest,
             choice_data = ChatCompletionResponseStreamChoice(
                 index=i, delta=DeltaMessage(role=role), finish_reason=None)
             chunk = ChatCompletionStreamResponse(id=request_id,
-                                                 object=obj_str,
+                                                 object=chunk_object_type,
                                                  created=created_time,
                                                  choices=[choice_data],
                                                  model=model_name)
@@ -258,7 +256,7 @@ async def create_chat_completion(request: ChatCompletionRequest,
                         delta=DeltaMessage(content=last_msg_content),
                         finish_reason=None)
                     chunk = ChatCompletionStreamResponse(id=request_id,
-                                                         object=obj_str,
+                                                         object=chunk_object_type,
                                                          created=created_time,
                                                          choices=[choice_data],
                                                          model=model_name)
@@ -273,8 +271,12 @@ async def create_chat_completion(request: ChatCompletionRequest,
             res: RequestOutput
             for output in res.outputs:
                 i = output.index
-                # Send token-by-token response for each request.n
-                if output.finish_reason is None and not finish_reason_sent[i]:
+
+                if finish_reason_sent[i]:
+                    continue
+
+                if output.finish_reason is None:
+                    # Send token-by-token response for each request.n
                     delta_text = output.text[len(previous_texts[i]):]
                     previous_texts[i] = output.text
                     completion_tokens = len(output.token_ids)
@@ -284,15 +286,14 @@ async def create_chat_completion(request: ChatCompletionRequest,
                         delta=DeltaMessage(content=delta_text),
                         finish_reason=None)
                     chunk = ChatCompletionStreamResponse(id=request_id,
-                                                         object=obj_str,
+                                                         object=chunk_object_type,
                                                          created=created_time,
                                                          choices=[choice_data],
                                                          model=model_name)
                     data = chunk.json(exclude_unset=True, ensure_ascii=False)
                     yield f"data: {data}\n\n"
-                # Send the finish response for each request.n only once
-                if output.finish_reason is not None and not finish_reason_sent[
-                        i]:
+                else:
+                    # Send the finish response for each request.n only once
                     prompt_tokens = len(res.prompt_token_ids)
                     final_usage = UsageInfo(
                         prompt_tokens=prompt_tokens,
@@ -302,7 +303,7 @@ async def create_chat_completion(request: ChatCompletionRequest,
                     choice_data = ChatCompletionResponseStreamChoice(
                         index=i, delta=[], finish_reason=output.finish_reason)
                     chunk = ChatCompletionStreamResponse(id=request_id,
-                                                         object=obj_str,
+                                                         object=chunk_object_type,
                                                          created=created_time,
                                                          choices=[choice_data],
                                                          model=model_name)
@@ -326,6 +327,7 @@ async def create_chat_completion(request: ChatCompletionRequest,
                                              "Client disconnected")
             final_res = res
         assert final_res is not None
+
         choices = []
         role = get_role()
         for output in final_res.outputs:
@@ -710,14 +712,8 @@ if __name__ == "__main__":
         engine_model_config.tokenizer,
         tokenizer_mode=engine_model_config.tokenizer_mode,
         trust_remote_code=engine_model_config.trust_remote_code)
+    load_chat_template(args, tokenizer)
 
-    chat_template = None
-    if args.chat_template is not None:
-        chat_template = load_chat_template()
-    if chat_template is not None:
-        tokenizer.chat_template = chat_template
-    if tokenizer.chat_template is not None:
-        logger.info(f"Using chat template:\n{tokenizer.chat_template}")
 
     uvicorn.run(app,
                 host=args.host,

[Tostino]

Thank you very much @simon-mo. I just added a handful of tests, and fixed the chatml template after tests identified some issues. Should be ready for you now.

• add_generation_prompt: is it really a request parameter or bound by the model itself. I have the feeling that most use case will always set it to one value and doesn't change across requests.

Agree here that most use cases will have a single value...but think of a chat UI that has two buttons (or hotkeys), one that sends the current message, and the other that has the model auto-complete the message the user is typing as they are typing. You would use different values for add_generation_prompt for each of those actions.

[simon-mo]

Oh one more future work could be loading chat template from http url. Let's see if that will be a common request and decide whether to be added.

[flexchar]

Hey all, I've been following the new and old thread for weeks, if not months, now. I wanted to tell that this is greatly awaited and appreciated work. Thank you @simon-mo and @Tostino for getting this into the framework.

Do we have an expected date to pin the new version for vLLM? :)

[Tostino]

@flexchar #1856

Should be pretty soon.

[flexchar]

I was waiting for this PR so long but I found out that making request to /chat/completion vs /completion where I manually prepare prepare ChatML string, ends up costing 2x as long. My input is 1000 tokens and output 100 tokens. @Tostino any ideas why that could be?

[Tostino]

I was waiting for this PR so long but I found out that making request to /chat/completion vs /completion where I manually prepare prepare ChatML string, ends up costing 2x as long. My input is 1000 tokens and output 100 tokens. @Tostino any ideas why that could be?

@flexchar Not off the top of my head, i've not had any noticeable slowdown between chat/completions and completions. I do notice a slowdown when using streaming (~10-20%), but nothing like a 2x slowdown.

I'm getting ~300-500 tokens/sec generated throughput using 1x 3090 and a llama2 13b AWQ model using the chat/completions endpoint.

Do you have an example you can share to trigger it?

Edit I tried it myself... could not reproduce. They both generate in roughly the same amount of time on my machine.

chat/completions:

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer " \
  -d '{
    "model": "Tostino/Inkbot-13B-8k-0.2",
    "stop": ["\n<#user#>", "\n<#bot#>"],
    "stream": false,
    "add_generation_prompt": true,
    "echo": false,
    "n": 1,
    "temperature": 0,
    "messages": [
	{"role": "meta-current_date", "content": "2023-10-20"},
	{"role": "meta-task_name", "content": "general"},
	{"role": "system", "content": "You are a helpful assistant. Please give long and detailed answers. You are an extremely competent electrical engineer."},
	{"role": "user", "content": "Hello!"},
	{"role": "assistant", "content": "Hello, how are you?"},
	{"role": "user", "content": "Please help me understand impedence matching in the context of a DC circuit."}
	]
  }'

completions:

curl http://0.0.0.0:8000/v1/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer " \
  -d '{
    "model": "Tostino/Inkbot-13B-8k-0.2",
    "echo": false,
    "max_tokens": 1024,
    "stream": false,
    "temperature": 0,
    "prompt": "<#meta#>\n- Date: 2023-10-20\n- Task: general\n<#system#>\nYou are a helpful assistant. Please give long and detailed answers. You are an extremely competent electrical engineer.\n<#chat#>\n<#user#>\nHello!\n<#bot#>\nHello, how are you?\n<#user#>\nPlease help me understand impedence matching in the context of a DC circuit.\n<#bot#>\n"
  }'

[flexchar]

Thank you for trying. I don't have an easy reproducable example but the next time I work on that part I will do one certainly. I appreciate you testing thou :)

[PeterXiaTian]

vllm/entrypoints/api_server.py　　　这里为什么还没有这个参数了？

[PhilipMay]

How do I use the chat template with the "offline inference" with vllm.LLM?

This PR only enables this for the REST API as far as I can see. @Tostino

[Tostino]

Sorry, on mobile right now so going from memory. I believe that there wasn't a "local" equivalent of the chat/completions API when I implemented this. So it was implemented for the REST endpoint, because that's all it could work with unless I did a bunch of extra work to also add an equivalent local version of the chat completions API.

[PeterXiaTian]

well，i see，thanks发自我的 iPhone在 2024年4月25日，23:11，Adam Brusselback ***@***.***> 写道： Sorry, on mobile right now so going from memory. I believe that there wasn't a "local" equivalent of the chat/completions API when I implemented this. So it was implemented for the REST endpoint, because that's all it could work with unless I did a bunch of extra work to also add an equivalent local version of the chat completions API. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>