[Doc]: `LLM.chat()` docstring incorrectly suggests multiple chats can be generated in one call

[capnrefsmmat]

📚 The doc issue

According to the docstring for the LLM class's chat() method,

messages – A list of messages to generate responses for. Each message is a list of dictionaries with ‘role’ and ‘content’ keys.

This implies messages can be a list containing multiple lists, each of which is one chat conversation with the dictionaries for role and content. However, that's not the case: messages must be a list of dictionaries, not a list of lists of dictionaries, because only one conversation is supported. See this example: #6416 (comment)

Suggest a potential alternative/fix

Two options:

1. The docstring is edited to say "A list of messages to generate responses for. Each message is a dictionary with ‘role’ and ‘content’ keys." Under Returns, say something like "A list containing one RequestOutput object with the generated response."
2. Make the function actually do what the docstring says. If you have multiple conversations you want responses to, doing them in a batch is much more efficient than doing them one at a time by repeatedly calling chat(). In my application, I need 12,000 chat responses, and calling chat() 12,000 times will take something like 12 hours, whereas calling generate() once with the pre-tokenized prompts takes an hour or two.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

I have updated the docstring in #8098 to indicate that the messages are only for a single conversation. If you have many conversations to process in batch, consider using run_batch.py.

[liuyanyi]

Hi, I also want to use LLM.chat to perform batch requests under my code, I verified it with a simple loop and it works.

    def chat(
        self,
        messages: List[ChatCompletionMessageParam], # TODO
        sampling_params: Optional[Union[SamplingParams,
                                        List[SamplingParams]]] = None,
        use_tqdm: bool = True,
        lora_request: Optional[LoRARequest] = None,
        chat_template: Optional[str] = None,
        add_generation_prompt: bool = True,
    ) -> List[RequestOutput]:

        tokenizer = self.get_tokenizer()
        model_config = self.llm_engine.get_model_config()

        # Check the first message to determine is the batch request 
        if not isinstance(messages[0], list):
            messages = [messages]

        batch_inputs = []
        for msg in messages:
            conversation, mm_data = parse_chat_messages(msg, model_config,
                                                        tokenizer)

            prompt = apply_chat_template(
                tokenizer,
                conversation,
                chat_template=chat_template,
                add_generation_prompt=add_generation_prompt,
            )

            inputs: PromptInputs
            if is_list_of(prompt, int):
                inputs = TokensPrompt(prompt_token_ids=prompt)
            else:
                inputs = TextPrompt(prompt=prompt)

            if mm_data is not None:
                inputs["multi_modal_data"] = mm_data
            
            batch_inputs.append(inputs)

        return self.generate(
            batch_inputs,
            sampling_params=sampling_params,
            use_tqdm=use_tqdm,
            lora_request=lora_request,
        )

results = vllm_model.chat(
            [
                [{"role": "user", "content": "Hello"}],
                [{"role": "user", "content": "Goodbye"}],
            ],
        )

[DarkLight1337]

Hi, I also want to use LLM.chat to perform batch requests under my code, I verified it with a simple loop and it works.

    def chat(
        self,
        messages: List[ChatCompletionMessageParam], # TODO
        sampling_params: Optional[Union[SamplingParams,
                                        List[SamplingParams]]] = None,
        use_tqdm: bool = True,
        lora_request: Optional[LoRARequest] = None,
        chat_template: Optional[str] = None,
        add_generation_prompt: bool = True,
    ) -> List[RequestOutput]:

        tokenizer = self.get_tokenizer()
        model_config = self.llm_engine.get_model_config()

        # Check the first message to determine is the batch request 
        if not isinstance(messages[0], list):
            messages = [messages]

        batch_inputs = []
        for msg in messages:
            conversation, mm_data = parse_chat_messages(msg, model_config,
                                                        tokenizer)

            prompt = apply_chat_template(
                tokenizer,
                conversation,
                chat_template=chat_template,
                add_generation_prompt=add_generation_prompt,
            )

            inputs: PromptInputs
            if is_list_of(prompt, int):
                inputs = TokensPrompt(prompt_token_ids=prompt)
            else:
                inputs = TextPrompt(prompt=prompt)

            if mm_data is not None:
                inputs["multi_modal_data"] = mm_data
            
            batch_inputs.append(inputs)

        return self.generate(
            batch_inputs,
            sampling_params=sampling_params,
            use_tqdm=use_tqdm,
            lora_request=lora_request,
        )

results = vllm_model.chat(
            [
                [{"role": "user", "content": "Hello"}],
                [{"role": "user", "content": "Goodbye"}],
            ],
        )

We have opened a new issue #8481 to track this.