Test Prompt Embeds/LoRA compatibility and Enable LoRA Support for OPT Models #25717
Conversation
Signed-off-by: Andrew Sansom <[email protected]>
Signed-off-by: Andrew Sansom <[email protected]>
qthequartermasterman
requested review from
DarkLight1337,
NickLucche,
aarnphm,
robertgshaw2-redhat and
simon-mo
as code owners
|
@DarkLight1337. I'm not sure who else would need to look at this. I also wonder that using this model could speed up some of the other entrypoints LoRA tests that are currently using zephyr-7b, just like you sped up these tests in #25663. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds LoRA support for OPT models and includes corresponding tests. The changes to enable LoRA in the OPT model implementation are mostly correct, following patterns from other models in the repository. However, I found a critical issue in the initialization of the LogitsProcessor which would lead to incorrect behavior when using LoRA adapters with extra vocabulary tokens. My review provides a code suggestion to fix this.
Signed-off-by: Andrew Sansom <[email protected]>
Signed-off-by: Andrew Sansom <[email protected]>
…list Signed-off-by: Andrew Sansom <[email protected]>
DarkLight1337
left a comment
DarkLight1337
left a comment
There was a problem hiding this comment.
LGTM if the tests pass, cc @jeejeelee if you want to double check the model
github-actions
bot
added
the
ready
cc @jeejeelee |
Signed-off-by: Andrew Sansom <[email protected]>
Signed-off-by: Andrew Sansom <[email protected]>
Head branch was pushed to by a user without write access
Signed-off-by: Andrew Sansom <[email protected]>
Signed-off-by: Andrew Sansom <[email protected]>
|
@DarkLight1337 This looks like it's ready for re-review. Thanks! Thanks @jeejeelee for your help. |
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <[email protected]>
… Models (#25717) Signed-off-by: Andrew Sansom <[email protected]> Signed-off-by: yewentao256 <[email protected]>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <[email protected]> Signed-off-by: Tomer Asida <[email protected]>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <[email protected]>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <[email protected]>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <[email protected]>
… Models (vllm-project#25717) Signed-off-by: Andrew Sansom <[email protected]>
3 participants
Purpose
It was previously unknown if #24278 was compatible with LoRA adapters or not. This PR adds tests explicitly for that combination. Since #25663 swapped out Zephyr for OPT125-m for testing prompt embeds, this PR also adds LoRA support for opt125-m.
Test Plan
Updated tests cases. I've also tested it locally with a meta-llama/Llama-3.1-8B-Instruct LoRA and everything seems to work as expected there.
Test Result
New tests are working locally. Pending CI.
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.