[V1] Add RayExecutor support for AsyncLLM (api server)
#11712
Conversation
jikunshang
requested review from
WoosukKwon,
robertgshaw2-neuralmagic,
njhill,
ywang96,
comaniac and
alexm-neuralmagic
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
Signed-off-by: Kunshang Ji <[email protected]>
| @@ -150,7 +151,11 @@ def _get_executor_cls(cls, vllm_config: VllmConfig) -> Type[Executor]: | |||
| executor_class: Type[Executor] | |||
| distributed_executor_backend = ( | |||
| vllm_config.parallel_config.distributed_executor_backend) | |||
| if distributed_executor_backend == "mp": | |||
| if distributed_executor_backend == "ray": | |||
There was a problem hiding this comment.
Instead of repeating this logic in both LLMEngine and AsyncLLM, can we instead move _get_executor_cls into abstract Executor? That way, we don't need to repeat logic across implementations. This is not a @classmethod anyways
There was a problem hiding this comment.
Good catch. Yeah we should unify them. cc @ruisearch42
| @@ -23,6 +23,7 @@ | |||
| from vllm.v1.engine.detokenizer import Detokenizer | |||
| from vllm.v1.engine.processor import Processor | |||
| from vllm.v1.executor.abstract import Executor | |||
| from vllm.v1.executor.ray_utils import initialize_ray_cluster | |||
There was a problem hiding this comment.
This should be a lazy import right? Doesn't this import ray?
There was a problem hiding this comment.
This will import Ray but won't error out if Ray is not installed:
try:
import ray
...
except ImportError:
ray = NoneIt errors out only when initialize_ray_cluster is called:
def initialize_ray_cluster():
assert_ray_available()
| @@ -150,7 +151,11 @@ def _get_executor_cls(cls, vllm_config: VllmConfig) -> Type[Executor]: | |||
| executor_class: Type[Executor] | |||
| distributed_executor_backend = ( | |||
| vllm_config.parallel_config.distributed_executor_backend) | |||
| if distributed_executor_backend == "mp": | |||
| if distributed_executor_backend == "ray": | |||
| initialize_ray_cluster(vllm_config.parallel_config) | |||
There was a problem hiding this comment.
This is called in RayExecutor constructor. No need to have it here.
|
Sorry forgot to disable auto-merge... @ruisearch42 could you submit a follow-up PR when you got a chance? Thanks |
Add
RayExecutorsupport forAsyncLLM(api server)