conversation template should come from huggingface tokenizer instead of fastchat

[ehartford]

vllm/entrypoints/openai/api_server.py

fastchat is a hack.
The real definition of conversation template is in the huggingface tokenizer.

https://huggingface.co/docs/transformers/main/chat_templating

Please use the huggingface tokenizer template instead of fastchat

>> from transformers import AutoTokenizer
>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")

>> chat = [
  {"role": "user", "content": "Hello, how are you?"},
  {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
  {"role": "user", "content": "I'd like to show off how chat templating works!"},
]

>> tokenizer.use_default_system_prompt = False
>> tokenizer.apply_chat_template(chat, tokenize=False)
"<s>[INST] Hello, how are you? [/INST] I'm doing great. How can I help you today? </s><s>[INST] I'd like to show off how chat templating works! [/INST]"

[Tostino]

Agreed. This is the right approach.

In this vein, I opened a PR for my hack for supporting custom templates: #1294
It still seems useful to be able to define the template for models that don't include it in the tokenizer, but it should probably be the same format that the HF prompt templates are defined in rather than copying the fastchat format.

[ehartford]

Absolutely! Very helpful to be able to specify it manually. Can this be set in an environment variable?

[Tostino]

Implemented: #1365

[Tostino]

Better implementation and some actual documentation: #1493

[hmellor]

Chat templates now come from HF