Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Model] Update multi-modal processor to support Mantis(LLaVA) model #10711

Merged
merged 24 commits into from
Dec 7, 2024
Merged

[Model] Update multi-modal processor to support Mantis(LLaVA) model #10711

merged 24 commits into from
Dec 7, 2024

Conversation

DarkLight1337
Copy link
Member

@DarkLight1337 DarkLight1337 commented Nov 27, 2024 •
edited by github-actions bot
Loading

This PR leverages the extensibility of the merged multi-modal processor to add support for Mantis (based on LLaVA), simply by overriding the HF processor to apply to the input data.

FIX #7143

@DarkLight1337
Add get_dummy_data to MultiModalProcessor; fix and test `iter_pla…
7b6c4f1 
…ceholders`

Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Use merged processor for llava model
de8332a 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
format
8b6804e 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Fix typo
26e3fdf 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Enable the test to pass on V1
93d27bc 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Handle embedding inputs
d697241 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
format
ca11cc9 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Merge branch 'main' into llava-mm-processor
c32cba9
@DarkLight1337
Fix wrong ndim
6c5c9ca 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Factor out merge_placeholders
0194324 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Fix placeholder maps handling on V0
09618d0 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Remove unused dummy data code
5501458 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Update dummy model
f3673c7 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Enable overriding hf processor and tokenizer; fix `_apply_prompt_repl…
37bc008 
…acements`

Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Improve error handling in _resolve_matches; merge matches directly
4805a9e 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Avoid hashing
8539008 
Signed-off-by: DarkLight1337 <[email protected]>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@DarkLight1337
Support and test Mantis model
1e82a4a 
Signed-off-by: DarkLight1337 <[email protected]>
@mergify mergify bot added ci/build documentation Improvements or additions to documentation labels Nov 27, 2024
This was referenced Nov 27, 2024
Open
Closed
@DarkLight1337
Update docs
cfbece4 
Signed-off-by: DarkLight1337 <[email protected]>
@mergify Mergify
Copy link

mergify bot commented Nov 29, 2024

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @DarkLight1337.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Nov 29, 2024
@mergify mergify bot removed the needs-rebase label Dec 7, 2024
@DarkLight1337
Fix type error
70c87d1 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337
Fix redundant code
27b276b 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337 DarkLight1337 marked this pull request as ready for review December 7, 2024 09:48
@DarkLight1337
Copy link
Member Author

DarkLight1337 commented Dec 7, 2024 •
edited
Loading

#10676 has been merged, so this PR is ready for review. The tests pass locally now.

Isotr0py
Isotr0py approved these changes Dec 7, 2024
View reviewed changes
Copy link
Collaborator

@Isotr0py Isotr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests also passed on my side. LGTM!

@Isotr0py Isotr0py enabled auto-merge (squash) December 7, 2024 14:47
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Dec 7, 2024
@DarkLight1337
Fix commands
d77cadd 
Signed-off-by: DarkLight1337 <[email protected]>
@DarkLight1337 DarkLight1337 mentioned this pull request Dec 7, 2024
Open
27 tasks
@DarkLight1337 DarkLight1337 changed the title [Model] Support Mantis(LLaVA) model [Model] Update multi-modal processor to support Mantis(LLaVA) model Dec 7, 2024
@DarkLight1337 DarkLight1337 enabled auto-merge (squash) December 7, 2024 17:03
@DarkLight1337 DarkLight1337 merged commit 39e227c into vllm-project:main Dec 7, 2024
76 checks passed
@DarkLight1337 DarkLight1337 deleted the mantis branch December 7, 2024 17:10
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
@DarkLight1337
[Model] Update multi-modal processor to support Mantis(LLaVA) model (v…
8581c85 
…llm-project#10711)

Signed-off-by: DarkLight1337 <[email protected]>
BKitor pushed a commit to BKitor/vllm that referenced this pull request Dec 30, 2024
@DarkLight1337 @BKitor
[Model] Update multi-modal processor to support Mantis(LLaVA) model (v…
1df386d 
…llm-project#10711)

Signed-off-by: DarkLight1337 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Isotr0py Isotr0py Isotr0py approved these changes

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

Assignees
No one assigned
Labels
ci/build documentation Improvements or additions to documentation ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[New Model]: Mantis
2 participants
@DarkLight1337 @Isotr0py