`OpenAI` endpoint using Templates for Chat Models

[flexchar]

Feature request

Hey, it'd be super nice to have the same endpoint when working with variety of libraries just like with Amazon S3.

Motivation

Many existing libraries support OpenAI as one of the endpoints. It's kinda of a must. However the open source models are becoming just as good to be a drop in replacement whether for custom fine tune, data privacy or other motivation.

I recently discovered that many fine tuned models include tokenizer with chat template. I even saw that on my fine tunes when inspecting tokenizer config after training with axolotl.
https://huggingface.co/docs/transformers/main/chat_templating

These are two offerings from HuggingFace and if they were to have a baby that would mean an OpenAI compatible endpoint like vLLM has, it'd be lit. 🔥

Example from vLLM:
https://vllm.readthedocs.io/en/latest/getting_started/quickstart.html#openai-compatible-server

This would allow HuggingFace models to be a complete drop-in replacement for customer without needing to ever open a terminal, code editor or server and spin up an endpoint using existing GUI for Spaces.

Your contribution

Let's RFC first. I'm afraid I don't have the expertise to contribute to Rust at this point. I'm sorry.

[OlivierDehaene]

You can use https://github.com/philschmid/easyllm for this.

[flexchar]

@OlivierDehaene thank for sharing that. Unfortunately it seems that is assuming that one is using Python. Personally I'm working with PHP back end, many are using other than Python as well. If it's beyond your scope, I'd totally understand!

[OlivierDehaene]

Ok I will keep this open but I don't know yet if we will add this to our roadmap.
Thanks for sharing!

[egeucak]

tokenizers contain the chat_template directly, so this should be possible https://huggingface.co/docs/transformers/v4.34.1/chat_templating

[Tostino]

Just FYI, I have a PR open for this feature with the vLLM inference server: vllm-project/vllm#1493

I've run a hundred million tokens through this implementation with a few different templates so far. All good.

Just noticed this thread, and them closing it saying "won't support". #1082
Quite disappointing.

[flexchar]

Guys, let's be understanding that this is open source software. Developer's time is expensive. They are not machines coding 24/7. HuggingFace already has several wonderful repositories such as this, ChatUI, transformers and all the wonderful things they have done to make things possible for each of us. Besides of that, they have to be a sustainable business.

I would love this just as much as anyone else. Running OpenAI equivalent endpoint using docker one liner and model name? Hell yes! But that said, it's not horribly hard to format the template until there's bandwidth to have this implemented. :)

[OlivierDehaene]

To use chat templates, you can simply hydrate the Jinja template in the tokenizer_config.json manually, through Transformers in Python and soon it will also be available in Transformers.js

We do not think it is a good idea to add it in TGI, the reason being that for a lot of cases, it does not provide enough flexibility. At the same time, we understand that running tokenization code client side can be a burden.
The door is not fully closed but that's where we stand today.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[flexchar]

✌️

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.