[Quantization] Quanto quantizer #29023
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
younesbelkada
left a comment
younesbelkada
left a comment
There was a problem hiding this comment.
Clean work already ! Looking forward to merge the PR !
|
cc @dacorvo |
|
Quick update, all tests are passing with the exception of safetensors tests. I've also implemented the quantization on the spot just like we do for bnb quantization. However, if the user wants to use cpu/disk offload, he will need to install the main branch of accelerate for now because of this PR. |
|
LGTM, mostly concerned with quanto specific function addition, that should leave in quanto.py not in the modeling (as much as possible of course). |
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
Thanks for iterating! 🔥 let's make quanto go brrr
| if hf_quantizer is not None: | ||
| missing_keys = hf_quantizer.update_missing_keys(model, missing_keys, prefix) |
Co-authored-by: Arthur <[email protected]>
…formers into quanto_integration
younesbelkada
left a comment
younesbelkada
left a comment
There was a problem hiding this comment.
Thanks again @SunMarc - great work ! 🚀 let's 🚢 it
|
Note to the core mainteners and especially @ArthurZucker . I reverted a lot of changes I did for the serialization. I decided to postpone the serialization feature by setting the I will do a follow up PR for the serialization since the PR to make it compatible with safetensors or Without serialization, this PR should still be good to merge ! |
5 participants
What does this PR do ?
This PR adds the quantization methods from quanto library. We will support inference + model quantization if the user perform weights only quantization since we don't require a calibration dataset.
TODO: