Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Misc][XPU] Avoid torch compile for XPU platform #10747

Merged
merged 4 commits into from
Dec 2, 2024
+8 −2
Merged

[Misc][XPU] Avoid torch compile for XPU platform #10747

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
5f75a72
Avoid torch compile for XPU platform
yma11 Nov 28, 2024
b38175b
add tp test for XPU
yma11 Nov 29, 2024
2e4e6a4
switch to use offline_inference_cli.py
yma11 Dec 2, 2024
29502ce
Update vllm/plugins/__init__.py
youkaichao Dec 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .buildkite/run-xpu-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,7 @@ remove_docker_container() { docker rm -f xpu-test || true; }
trap remove_docker_container EXIT
remove_docker_container

# Run the image and launch offline inference
docker run --network host --name xpu-test --device /dev/dri -v /dev/dri/by-path:/dev/dri/by-path --entrypoint="" xpu-test python3 examples/offline_inference.py
# Run the image and test offline inference/tensor parallel
docker run -it -d --name xpu-test --device /dev/dri -v /dev/dri/by-path:/dev/dri/by-path xpu-test /bin/bash
docker exec xpu-test bash -c "python3 examples/offline_inference.py"
docker exec xpu-test bash -c "python3 examples/offline_inference_cli.py -tp 2"
3 changes: 3 additions & 0 deletions vllm/plugins/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import torch

import vllm.envs as envs
from vllm.platforms import current_platform

logger = logging.getLogger(__name__)

Expand All @@ -25,6 +26,8 @@ def load_general_plugins():
os.environ['TORCHINDUCTOR_COMPILE_THREADS'] = '1'
# see https://github.com/vllm-project/vllm/issues/10619
torch._inductor.config.compile_threads = 1
if current_platform.is_xpu():
os.environ['TORCH_COMPILE_DISABLE'] = 'True'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any documentation for this env var?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean where this config comes from? If so, please refer here.

youkaichao marked this conversation as resolved.
Show resolved Hide resolved
global plugins_loaded
if plugins_loaded:
return
Expand Down