add shared experts for upcoming Granite 4.0 language models #35894
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@ArthurZucker can you merge this? |
There was a problem hiding this comment.
Hey! As always for us this will be a new model, with modular it should be super easy to add however! https://huggingface.co/docs/transformers/en/modular_transformers 🤗
|
We're adding an additional feature (shared experts) that doesn't break past checkpoints, and is an extension of our own model class. Would every extension entail a new model class? |
|
Yes 🤗 I am sorry but that is the way we have been handling every single model so far! |
shawntan
force-pushed
the
shared-experts
branch
from
a0fd52a to
9b2652c
Compare
shawntan
force-pushed
the
shared-experts
branch
from
c09ee2f to
d81c969
Compare
|
Not sure how to get the tests to pass , some are not due to the changes I've made. |
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Signed-off-by: Sukriti-Sharma4 <[email protected]>
Signed-off-by: Sukriti-Sharma4 <[email protected]>
|
@ArthurZucker the PR is ready, please review. |
ArthurZucker
approved these changes
Feb 13, 2025
|
|
||
|
|
||
| class GraniteMoeSharedForCausalLM(GraniteMoeForCausalLM): | ||
| _tied_weights_keys = ["lm_head.weight"] |
There was a problem hiding this comment.
if the mlp is shared, should it appear here?
There was a problem hiding this comment.
no, it shouldnt.
shared means its a shared in sense of experts (not across layers)
|
Thanks for approving, please merge as soon as possible :) |
Signed-off-by: Sukriti-Sharma4 <[email protected]>
|
I have updated with the main branch , made corresponding changes and all checks have passed :) |
ArthurZucker
pushed a commit
that referenced
this pull request
Feb 17, 2025
* Modular GraniteMoE with shared Experts. Signed-off-by: Shawn Tan <[email protected]> * Modified * Import order. * Modified for style * Fix space. * Test * Remove extra granitemoe file. * New converted file and tests * Modified __init__ files. * Formatting. * Dummy PT objects * register granitemoe shared model Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix linting of a file Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix import in modeling file Signed-off-by: Sukriti-Sharma4 <[email protected]> * update generated modeling file Signed-off-by: Sukriti-Sharma4 <[email protected]> * add documentation Signed-off-by: Sukriti-Sharma4 <[email protected]> * update docstrings Signed-off-by: Sukriti-Sharma4 <[email protected]> * update generated modeling file Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix docstrings in config class Signed-off-by: Sukriti-Sharma4 <[email protected]> * merge main Signed-off-by: Sukriti-Sharma4 <[email protected]> --------- Signed-off-by: Shawn Tan <[email protected]> Signed-off-by: Sukriti-Sharma4 <[email protected]> Co-authored-by: Shawn Tan <[email protected]> Co-authored-by: Shawn Tan <[email protected]> Co-authored-by: Sukriti-Sharma4 <[email protected]> Co-authored-by: Sukriti Sharma <[email protected]>
zucchini-nlp
pushed a commit
to zucchini-nlp/transformers
that referenced
this pull request
Feb 21, 2025
…ace#35894) * Modular GraniteMoE with shared Experts. Signed-off-by: Shawn Tan <[email protected]> * Modified * Import order. * Modified for style * Fix space. * Test * Remove extra granitemoe file. * New converted file and tests * Modified __init__ files. * Formatting. * Dummy PT objects * register granitemoe shared model Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix linting of a file Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix import in modeling file Signed-off-by: Sukriti-Sharma4 <[email protected]> * update generated modeling file Signed-off-by: Sukriti-Sharma4 <[email protected]> * add documentation Signed-off-by: Sukriti-Sharma4 <[email protected]> * update docstrings Signed-off-by: Sukriti-Sharma4 <[email protected]> * update generated modeling file Signed-off-by: Sukriti-Sharma4 <[email protected]> * fix docstrings in config class Signed-off-by: Sukriti-Sharma4 <[email protected]> * merge main Signed-off-by: Sukriti-Sharma4 <[email protected]> --------- Signed-off-by: Shawn Tan <[email protected]> Signed-off-by: Sukriti-Sharma4 <[email protected]> Co-authored-by: Shawn Tan <[email protected]> Co-authored-by: Shawn Tan <[email protected]> Co-authored-by: Sukriti-Sharma4 <[email protected]> Co-authored-by: Sukriti Sharma <[email protected]>
|
Is the public release of Granite-4.0-Tiny-Preview in any way relevant to this PR? (like does it warrant any follow-up work, additional validation/testing/CI, etc.) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for shared experts in GraniteMoE model class for upcoming Granite 4.0 language models.
@ArthurZucker