[Core] Failure in pytorch distributed training code failed to get a job into FAILED state

[Michaelvll]

A user reported that although their distributed training code failed due to NCCL error on worker node, SkyPilot does not set the job in FAILED state.

Version & Commit info:

• sky -v: PLEASE_FILL_IN
• sky -c: PLEASE_FILL_IN

[Michaelvll]

It turns out that the user program has an 0 return code even if a failure happens, which causes our driver process to wait for any in-progress processes to return with 0 return code.