Skip to content

Conversation

@hanouticelina
Copy link
Contributor

Related to #2720.
logit_bias is not implemented in TGI (see the comment here), this PR updates the parameter documentation to flag it as unused.

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@juanwisz
Copy link

Could we consider actually implementing logit_bias? It's a very valuable feature for controlling model outputs, I've opened in TGI to discuss implementation details: huggingface/text-generation-inference#2869

@hanouticelina
Copy link
Contributor Author

Could we consider actually implementing logit_bias? It's a very valuable feature for controlling model outputs, I've opened in TGI to discuss implementation details: huggingface/text-generation-inference#2869

@juanwisz This PR only aligns the client documentation with the current TGI implementation. Once/If logit_bias is implemented on TGI side, we'll update the documentation accordingly 🤗

@LysandreJik
Copy link
Member

Is it possible to remove that doc attribute altogether? If not, I would at least specify why it's unused and how to have users request for this argument to be used

@hanouticelina
Copy link
Contributor Author

Is it possible to remove that doc attribute altogether? If not, I would at least specify why it's unused and how to have users request for this argument to be used

@LysandreJik logit_bias and n (i.e number of chat completion choices to generate for each input message) which is already flagged as unused, are implemented in the OpenAI API, and just like we expose these parameters in TGI to maintain OpenAI compatibility, we want to keep them in the client for the same reason —making InferenceClient a proper drop-in replacements for OpenAI client. I've updated their documentation to indicate they are currently not implemented in TGI while explaining why they are still included.

Copy link
Member

@LysandreJik LysandreJik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok great! Thank you Célina !

@hanouticelina hanouticelina merged commit 6f5d870 into main Jan 6, 2025
17 checks passed
@hanouticelina hanouticelina deleted the update-logit-bias-documentation branch January 6, 2025 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants