[Feature Request] Make v1/completions behave as OpenAI API

[comaniac]

🚀 Feature

The current v1/completions in REST API (mlc_chat/rest.py:142) implementation only takes prompt and returns the generated texts. However, it should considers all parameters in the request (e.g., temperature, etc). Also, the current response only includes the generated text, it would be better to also return other information as well as logprobs (if requested).

Motivation

We are evaluating open source LLMs and would like to compare quantized LLaMA 7B model against the BF16 model on:

1. The quality by HELM evaluation, and
2. The memory and latency

Alternatives

N/A

Additional context

N/A

[junrushao]

Having post-softmax scores is a critical step towards HELM evaluation of LLMs. In MLC LLM, we may call softmax_with_temperature in the generated DSO to obtain the softmax score, but it's probably not exposed in our OpenAI API yet. CC @sudeepag if you are interested :-)

[sudeepag]

Yes, it has been on my todo list for a while 😄, I'll look into adding support.

[lhl]

Just to add onto this, I was trying to test this out including a "role": "system" message and got the following error:

  File "/home/local/.conda/envs/omniquant/lib/python3.10/site-packages/mlc_chat/rest.py", line 156, in request_completion
    raise ValueError(
ValueError: 
                The /v1/chat/completions endpoint currently only supports single message prompts.
                Please ensure your request contains only one message

Sadly, this makes the mlc_chat.rest module basically incompatible with any OpenAI API client or use case involving conversations or instructions?

[davidpissarra]

MLC LLM is now compatible with OpenAI parameters #1107

[comaniac]

Hey @davidpissarra thanks for the PR. I notice that although most features are covered, logprob is still missing. Is the logprobs support not considered at this moment?

[tqchen]

Thanks @comaniac for the suggestion.

@Kartik14 @davidpissarra @MasterJH5574 would be great for us to followup with logprob

[Kartik14]

Yes, we will take a look at this.