Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Misc][LoRA] Clean up the function interface of Punica #10917

Merged
merged 27 commits into from
Dec 5, 2024

Conversation

jeejeelee
Copy link
Collaborator

@jeejeelee jeejeelee commented Dec 5, 2024

Description

The current Punica interfaces(functions whose names start with add_ in punica.py) are somewhat complex and disorganized. This PR primarily categorizes all LoRA layers into the following three types and clean up related interfaces. The benefits include simplifying and standardizing functions to facilitate LoRA support across other platforms, such as: #4830 #10565

1. Linear

Three functions are provided, where add_lora_linear is for all linear lora in layers.py, while add_shrink and add_expand are designed for full sharded lora in fully_sharded_layers.py

2. Embedding

Only one function is needed, see: add_lora_embedding

3. Logits

Only one function is needed,see:add_lora_logits

cc @SanjuCSudhakaran
ping @DarkLight1337
@Isotr0py Since you're the author of #4830, would you please take a look at this PR as well?

Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Copy link

github-actions bot commented Dec 5, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Copy link
Collaborator

@Isotr0py Isotr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for cleaning up the duplicate function! I added some initial comments about the punica interface. PTAL!

vllm/lora/layers.py Outdated Show resolved Hide resolved
vllm/lora/punica.py Outdated Show resolved Hide resolved
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
@jeejeelee jeejeelee requested a review from Isotr0py December 5, 2024 09:04
Copy link
Collaborator

@Isotr0py Isotr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM!

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) December 5, 2024 10:40
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Dec 5, 2024
@DarkLight1337 DarkLight1337 merged commit 571da8f into vllm-project:main Dec 5, 2024
66 checks passed
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
BKitor pushed a commit to BKitor/vllm that referenced this pull request Dec 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants