Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Misc] Enhance offline_inference to support user-configurable paramet… #10392

Merged
merged 6 commits into from
Nov 17, 2024
Merged

[Misc] Enhance offline_inference to support user-configurable paramet… #10392

merged 6 commits into from
Nov 17, 2024

Conversation

wchen61
Copy link
Contributor

@wchen61 wchen61 commented Nov 16, 2024 •
edited by github-actions bot
Loading

FIX #10391

@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@wchen61 wchen61 force-pushed the enhance_offline_inference branch from 603f180 to 3c27e97 Compare November 16, 2024 16:01
Isotr0py
Isotr0py reviewed Nov 16, 2024
View reviewed changes
Copy link
Collaborator

@Isotr0py Isotr0py left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this enhancement! However, since this is an example script used to introduce the usage of LLM entrypoint for offline inference, we should keep it clean and simple as much as possible.

Therefore, we should expose arguments that are in common use or highlights in vLLM (like tensor parallel, chunked prefill and prefix caching etc).

examples/offline_inference.py Outdated Show resolved Hide resolved
examples/offline_inference.py Outdated Show resolved Hide resolved
examples/offline_inference.py Outdated Show resolved Hide resolved
examples/offline_inference.py Show resolved Hide resolved
@wchen61 wchen61 force-pushed the enhance_offline_inference branch from eab5c7b to 28d6d60 Compare November 17, 2024 02:36
Isotr0py
Isotr0py reviewed Nov 17, 2024
View reviewed changes
Copy link
Collaborator

@Isotr0py Isotr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, just leave some nits.

examples/offline_inference.py Outdated Show resolved Hide resolved
examples/offline_inference.py Outdated Show resolved Hide resolved
Isotr0py
Isotr0py approved these changes Nov 17, 2024
View reviewed changes
Copy link
Collaborator

@Isotr0py Isotr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Isotr0py Isotr0py enabled auto-merge (squash) November 17, 2024 10:45
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 17, 2024
@Isotr0py Isotr0py merged commit d1557e6 into vllm-project:main Nov 17, 2024
41 of 43 checks passed
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Nov 18, 2024
@wchen61
[Misc] Enhance offline_inference to support user-configurable paramet… (
242bb53 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
coolkp pushed a commit to coolkp/vllm that referenced this pull request Nov 20, 2024
@wchen61 @coolkp
[Misc] Enhance offline_inference to support user-configurable paramet… (
d4dc0fb 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024
@wchen61
[Misc] Enhance offline_inference to support user-configurable paramet… (
39f9cf5 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
mfournioux pushed a commit to mfournioux/vllm that referenced this pull request Nov 20, 2024
@wchen61 @mfournioux
[Misc] Enhance offline_inference to support user-configurable paramet… (
dc8d45f 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
Signed-off-by: Maxime Fournioux <[email protected]>
rickyyx pushed a commit to rickyyx/vllm that referenced this pull request Nov 20, 2024
@wchen61 @rickyyx
[Misc] Enhance offline_inference to support user-configurable paramet… (
48443fb 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
Signed-off-by: rickyx <[email protected]>
@WoosukKwon WoosukKwon mentioned this pull request Nov 21, 2024
Merged
tlrmchlsmth pushed a commit to neuralmagic/vllm that referenced this pull request Nov 23, 2024
@wchen61 @tlrmchlsmth
[Misc] Enhance offline_inference to support user-configurable paramet… (
52d676e 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
Signed-off-by: Tyler Michael Smith <[email protected]>
prashantgupta24 pushed a commit to opendatahub-io/vllm that referenced this pull request Dec 3, 2024
@wchen61 @prashantgupta24
[Misc] Enhance offline_inference to support user-configurable paramet… (
d97d269 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
@wchen61
[Misc] Enhance offline_inference to support user-configurable paramet… (
fb62568 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Isotr0py Isotr0py Isotr0py approved these changes

Assignees
No one assigned
Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature]: Enhance offline_inference.py with Configurable Parameters for Greater Flexibility
2 participants
@wchen61 @Isotr0py