Skip to content

Commit

Permalink
Add step to fix NCCL error with default environment
Browse files Browse the repository at this point in the history
  • Loading branch information
XkunW authored May 23, 2024
1 parent 8d7c593 commit 3ec9069
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
This repository provides an easy-to-use solution to run inference servers on [Slurm](https://slurm.schedmd.com/overview.html)-managed computing clusters using [vLLM](https://docs.vllm.ai/en/latest/). All scripts in this repository runs natively on the Vector Institute cluster environment, and can be easily adapted to other environments.

## Installation
If you are using the Vector cluster environment, and you don't need any customization to the inference server environment, you can skip this step and go to the next section. Otherwise, you might need up to 10GB of storage to setup your own virtual environment. The following steps needs to be run only once for each user.
If you are using the Vector cluster environment, and you don't need any customization to the inference server environment, all you need to do is run `pip install vllm-nccl-cu12` and go to the next section. Otherwise, you might need up to 10GB of storage to setup your own virtual environment. The following steps needs to be run only once for each user.

1. Setup the virtual environment for running inference servers, run
```bash
Expand Down

0 comments on commit 3ec9069

Please sign in to comment.