Skip to content

feat: add chat template to existed gguf file #6156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

feat: add chat template to existed gguf file #6156

wants to merge 1 commit into from

Conversation

bruceunx
Copy link

feat: add chat template to existed gguf file

follow the issue 5897

add gguf_add_chat_template.py to gguf-py/scripts

feat: add chat template to existed gguf file
@ngxson
Copy link
Collaborator

ngxson commented Mar 19, 2024

Thanks for the contribution. However, I'm quite doubt if this file may be confused for new users. Some users may misunderstand that this script allow them to use which ever templates they want, without not knowing that llama.cp can only support a predefine set of templates

I think it would be nice to include some kind of warning. And would be even nicer if you can take time to allow the script to change other attributes in metadata.

@bruceunx
Copy link
Author

bruceunx commented Mar 20, 2024

the chat template actually trained in the model, and normally included in the tokenizer_config.json in original model file, while some old gguf missed the chat template in metadata while quantized.
Actually It's not proper to add any other template to model which was not trained in the model.
I thought this script just for gguf which quantized before without including chat template in it, this gguf file is all OK just missing the proper chat template. :)

@bruceunx
Copy link
Author

bruceunx commented Mar 20, 2024

Maybe I can add predefined chat templates for known popular open source llms models, which maybe solve this concern

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants