[Feature] Allow easily overriding command for head/worker pods

[kwohlfahrt]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

I would like to override just the command part of the head/worker pods of a cluster. Specifically, right now I would like to use bash -c instead of bash -lc, but I could also see situations where I want to use a different shell (e.g. if bash is not present in my image).

However, if I set command: ["bash", "-c"] on the cluster template, I end up with the pod having the specification below, which is not what I want:

    Command:
      /bin/bash
      -lc
      --
    Args:
       bash  -c  && ulimit -n 65536; ray start --head  --dashboard-host=0.0.0.0  --metrics-export-port=8080  --block  --dashboard-agent-listen-port=52365  --no-monitor  --num-cpus=1  --memory=1073741824 

If I manually specify both command and args, this works for the head pod. However, this becomes difficult for workers, because the wait-gcs-ready args are quite complex.

Use case

I have a Ray image based on micromamba, which sets $PATH in login shells (in /etc/profile). Therefore, the default command of bash -lc doesn't work (it cannot find the ray binary). Running a normal shell (with bash -c) works fine.

FWIW, I don't think using a login shell by default is correct, because this isn't an interactive login. But I see it might be difficult to change the default for compatiblity reasons, so just making it easily overridable would be fine.

Related issues

#2208 is related, but is just for the submitter pod of jobs, while I want to do the same for the head & worker pods.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!