[Feature]: ChatCompletionRequest get default value from generation_config.json

[zhaotyer]

🚀 The feature, motivation and pitch

temperature/top_k/top_p These values ​​will affect the model output，The default value should be read from generation_config.json if the user does not set it.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[youkaichao]

Why would this be important? We do not read generation_config.json right now, and we don't see many complaints. If we change to this behavior, it will break backward compatibility, and affect many users.

[yansh97]

A few days ago, I also noticed it and submitted an issue. #10539

If we don’t consider beam-search, there are actually only five parameters worth extracting from generation_config.json to be used as arguments for SamplingParams. They are {"repetition_penalty", "temperature", "top_k", "top_p", "min_p"}.

Currently, I’m using an ASGI middleware to automatically add default values into request body in the OpenAI API server. Below is the implementation of the middleware:

from typing import Any
from starlette.types import ASGIApp, Message, Receive, Scope, Send
from transformers import GenerationConfig

def _create_sampling_params(*, model_id: str) -> dict[str, Any]:
    fields: set[str] = {"repetition_penalty", "temperature", "top_k", "top_p", "min_p"}
    try:
        config: dict[str, Any] = GenerationConfig.from_pretrained(
            pretrained_model_name=model_id
        ).to_diff_dict()
    except Exception:
        config = {}
    return {k: config[k] for k in sorted(config.keys()) if k in fields}


def _create_sampling_params_middleware(*, model_id: str) -> type | None:
    sampling_params: dict[str, Any] = _create_sampling_params(model_id=model_id)
    if not sampling_params:
        return None

    class SamplingParamsMiddleware:
        def __init__(self, app: ASGIApp) -> None:
            self.app = app

        async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
            if (
                scope["type"] != "http"
                or scope["method"] != "POST"
                or scope["path"] != "/v1/chat/completions"
            ):
                await self.app(scope, receive, send)
                return

            async def modify_receive() -> Message:
                message = await receive()
                if message["type"] != "http.request":
                    return message
                assert message["more_body"] is False
                assert "body" in message
                body: dict[str, Any] = sampling_params | json.loads(
                    s=message["body"].decode()
                )
                message["body"] = json.dumps(obj=body).encode()
                return message

            await self.app(scope, modify_receive, send)

    return SamplingParamsMiddleware

I believe it’s necessary to provide a CLI parameter to enable or disable the default values from generation_config.json. Otherwise, it could force some users to hack the vLLM code. @youkaichao @DarkLight1337

[youkaichao]

adding an cli arg to make this an engine level flag makes sense to me. it should be an opt-in feature without breaking backward compatibility.

cc @DarkLight1337

[chaunceyjiang]

Please assign to me.

[liuyanyi]

Hi, i'm interested in this feature, and i make a workable solution for openai server.

I managed to modify the Pydantic model to overwrite the default sampling param, and swagger doc can also show the modify. How do you guys think of this implementation.

You can have a look at here.
main...liuyanyi:vllm:add_generation_config

[yansh97]

Hi, i'm interested in this feature, and i make a workable solution for openai server.你好，我对这个功能很感兴趣，我为 openai 服务器做了一个可行的解决方案。

I managed to modify the Pydantic model to overwrite the default sampling param, and swagger doc can also show the modify. How do you guys think of this implementation.我设法修改 Pydantic 模型以覆盖默认采样参数，并且 swagger doc 也可以显示修改。大家觉得这个实现怎么样？

You can have a look at here.你可以在这里看看。 main...liuyanyi:vllm:add_generation_config

I think this should be a parameter for the Engine class, not for the OpenAI server.

[liuyanyi]

It's true that introducing to engine_args would be better, but I don't have much of a clue on the implementation

My implementation is meant to be as simple as possible, right now I'm using the model mainly through the LLM class and the OpenAI server, whereas the LLM class can be used to pass in the sampling param directly from the outside, so I think starting with the server would be easier.

[yansh97]

It's true that introducing to engine_args would be better, but I don't have much of a clue on the implementation确实，引入engine_args会更好，但我对实现没有太多线索

My implementation is meant to be as simple as possible, right now I'm using the model mainly through the LLM class and the OpenAI server, whereas the LLM class can be used to pass in the sampling param directly from the outside, so I think starting with the server would be easier.我的实现是为了尽可能简单，现在我主要通过LLM类和OpenAI服务器使用模型，而LLM类可以用于直接从外部传入采样参数，所以我认为从服务器开始会更容易。

You can also add sampling parameters in the body of the request. I believe it is more important to maintain consistency between the LLM class and the OpenAI server.