Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Idefics2 #1270

Merged
merged 14 commits into from
Nov 7, 2024
Merged

Idefics2 #1270

merged 14 commits into from
Nov 7, 2024

Conversation

sywangyi
Copy link
Collaborator

What does this PR do?

Fixes # (issue)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@sywangyi
add idefics2
4fb058a 
Signed-off-by: Wang, Yi A <[email protected]>
@sywangyi
add example and test
b5beca4 
Signed-off-by: Wang, Yi A <[email protected]>
@sywangyi
fix readme
a3890a1 
Signed-off-by: Wang, Yi A <[email protected]>
@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@sywangyi
Copy link
Collaborator Author

sywangyi commented Aug 19, 2024 •
edited
Loading

throught(tokens/s) A100 Gaudi2
BF16 11.7 53.6
FP32 15.3 29.53

cmd:
python3 run_pipeline.py --model_name_or_path HuggingFaceM4/idefics2-8b --use_hpu_graphs --bf16

ssarkar2
ssarkar2 reviewed Sep 5, 2024
View reviewed changes
Copy link
Collaborator

@ssarkar2 ssarkar2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good otherwise, have a question regd unfold

@sywangyi
image2text finetune
13f5f77 
Signed-off-by: Wang, Yi A <[email protected]>
@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 6, 2024

The code quality check failed, please run make style.

@sywangyi
format fix
21ace73 
Signed-off-by: Wang, Yi A <[email protected]>
@sywangyi
fix generate problem, eos is incorrect handled if batch_size > 1
ad61736 
move vision model to prepare input(image size may change if the input is dataset, can not use hpu graph to optimize it)
add ci test

Signed-off-by: Wang, Yi A <[email protected]>
@sywangyi
update doc
b6b2bae 
Signed-off-by: Wang, Yi A <[email protected]>
@sywangyi
fix doc
998574e 
Signed-off-by: Wang, Yi A <[email protected]>
@sywangyi
add static shape support in image process, replace unfold with conv2d…
960b730 
… to speedup finetune

Signed-off-by: Wang, Yi A <[email protected]>
@sywangyi
Copy link
Collaborator Author

Looks good otherwise, have a question regd unfold

Hi, I have replace unfold with conv2D. cpu fallback is not needed. finetune got speedup @ssarkar2

@sywangyi
Copy link
Collaborator Author

has found the recent PR which upgrade test_image_to_text_example.py. add "use_kv_cache", whose default value is false if not indicated explicitly in command line. which will lead image-to-text ci test to fail. fix it.

@sywangyi
Copy link
Collaborator Author

also found the similar PR which mean to fix eos issue if batch>1, please refer #1287,
the root cause of it is the needs_tensor_output is incorrect set

@ssarkar2 ssarkar2 mentioned this pull request Oct 11, 2024
Closed
3 tasks
@libinta libinta added run-test Run CI for PRs from external contributors and removed review wip labels Oct 15, 2024
@sywangyi
Copy link
Collaborator Author

sywangyi commented Nov 7, 2024

@regisss could you review and merge it?

@regisss regisss merged commit 7ac1db1 into main Nov 7, 2024
7 checks passed
@regisss regisss deleted the idefics2 branch November 7, 2024 20:26
Luca-Calabria pushed a commit to Luca-Calabria/optimum-habana that referenced this pull request Nov 21, 2024
@sywangyi @Luca-Calabria
Idefics2 (huggingface#1270)
b4dd264 
Signed-off-by: Wang, Yi A <[email protected]>
Luca-Calabria pushed a commit to Luca-Calabria/optimum-habana that referenced this pull request Nov 25, 2024
@sywangyi @Luca-Calabria
Idefics2 (huggingface#1270)
a9bb186 
Signed-off-by: Wang, Yi A <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ssarkar2 ssarkar2 ssarkar2 approved these changes

@regisss regisss regisss approved these changes

@bhargaveede bhargaveede Awaiting requested review from bhargaveede bhargaveede is a code owner

@vivekgoe vivekgoe Awaiting requested review from vivekgoe vivekgoe is a code owner

Assignees
No one assigned
Labels
run-test Run CI for PRs from external contributors
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@sywangyi @HuggingFaceDocBuilderDev @regisss @ssarkar2 @libinta