[RayCluster][Feature] add GcsFaultToleranceOptions to the RayCluster CRD #2712

rueian · 2025-01-03T20:21:55Z

Why are these changes needed?

Currently, we need users to set the ray.io/ft-enabled annotation, their redis address in envs and their redis password in both envs and RayStartParams in their RayCluster CR to enable GCS FT. Too many places to set up.

This PR adds GcsFaultToleranceOptions to the RayCluster CRD, unifying all the GCS FT settings in one field to provide a better user experience and all required annotations, envs, and ray start parameters will be set according to this new GcsFaultToleranceOptions.

With the new GcsFaultToleranceOptions, a GCS FT enabled RayCluster CR will look like this:

---
apiVersion: v1
kind: Secret
metadata:
  name: redis-password-secret
type: Opaque
data:
  # echo -n "5241590000000000" | base64
  password: NTI0MTU5MDAwMDAwMDAwMA==
---
apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: raycluster-external-redis
spec:
  rayVersion: '2.9.0'
  gcsFaultToleranceOptions:
    redisAddress: redis:6379
    redisPassword:
      valueFrom:
        secretKeyRef:
          name: redis-password-secret
          key: password
  headGroupSpec:
    rayStartParams: {}
    template:
      spec:
        containers:
          - name: ray-head
            image: rayproject/ray:2.9.0-aarch64
            resources:
              limits:
                cpu: "1"
              requests:
                cpu: "1"
            ports:
              - containerPort: 6379
                name: redis
              - containerPort: 8265
                name: dashboard
              - containerPort: 10001
                name: client

Related issue number

Resolves #2695

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: Rueian <[email protected]>

…tions Signed-off-by: Rueian <[email protected]>

Signed-off-by: Rueian <[email protected]>

…edis-password Signed-off-by: Rueian <[email protected]>

rueian · 2025-01-04T17:20:17Z

ray-operator/controllers/ray/common/pod.go

@@ -96,6 +96,9 @@ func initTemplateAnnotations(instance rayv1.RayCluster, podTemplate *corev1.PodT
 			podTemplate.Annotations[utils.RayExternalStorageNSAnnotationKey] = v
 		}
 	}
+	if options := instance.Spec.GcsFaultToleranceOptions; options != nil && options.ExternalStorageNamespace != "" {
+		podTemplate.Annotations[utils.RayExternalStorageNSAnnotationKey] = options.ExternalStorageNamespace


Override the ray.io/external-storage-namespace annotation if the new ExternalStorageNamespace is specified. We rely on this annotation to set the RAY_external_storage_namespace env in the later setContainerEnvVars function.

rueian · 2025-01-04T17:21:23Z

ray-operator/controllers/ray/common/pod.go

@@ -322,7 +325,7 @@ func initLivenessAndReadinessProbe(rayContainer *corev1.Container, rayNodeType r
 }

 // BuildPod a pod config
-func BuildPod(ctx context.Context, podTemplateSpec corev1.PodTemplateSpec, rayNodeType rayv1.RayNodeType, rayStartParams map[string]string, headPort string, enableRayAutoscaler *bool, creatorCRDType utils.CRDType, fqdnRayIP string) (aPod corev1.Pod) {
+func BuildPod(ctx context.Context, podTemplateSpec corev1.PodTemplateSpec, rayNodeType rayv1.RayNodeType, gcsOptions *rayv1.GcsFaultToleranceOptions, rayStartParams map[string]string, headPort string, enableRayAutoscaler *bool, creatorCRDType utils.CRDType, fqdnRayIP string) (aPod corev1.Pod) {


Just add a new gcsOptions parameter and pass it to the later setContainerEnvVars function.

rueian · 2025-01-04T18:15:59Z

ray-operator/controllers/ray/common/pod.go

+	if rayNodeType == rayv1.HeadNode && gcsOptions != nil {
+		if gcsOptions.RedisUsername != nil {
+			rayStartParams["redis-username"] = "$REDIS_USERNAME"
+		}
+		if gcsOptions.RedisPassword != nil {
+			rayStartParams["redis-password"] = "$REDIS_PASSWORD"
+		}
+	}


If the new gcsOptions is specified with redis username and password, we add --redis-username and --redis-password flags to the ray start command.

The $REDIS_USERNAME and $REDIS_PASSWORD will be replaced with env vars by the shell.

rueian · 2025-01-04T18:17:26Z

ray-operator/controllers/ray/common/pod.go

+	if !utils.EnvVarExists(utils.REDIS_USERNAME, container.Env) {
+		// setting the REDIS_USERNAME env var from the params
+		if value, ok := rayStartParams["redis-username"]; ok {
+			container.Env = append(container.Env, corev1.EnvVar{Name: utils.REDIS_USERNAME, Value: value})
+		}
+	}


Although the support for Redis usernames in the Ray core has been merged but hasn't been released yet, I think we can add related code in KubeRay 1.3.

rueian · 2025-01-04T18:19:51Z

ray-operator/controllers/ray/raycluster_controller.go

+	if utils.EnvVarExists(utils.REDIS_USERNAME, pod.Spec.Containers[utils.RayContainerIndex].Env) {
+		pod.Spec.Containers[utils.RayContainerIndex].Args[0] += "sys.exit(1) if not cleanup_redis_storage(host=parsed.hostname, port=parsed.port, username=os.getenv('REDIS_USERNAME', parsed.username), password=os.getenv('REDIS_PASSWORD', parsed.password), use_ssl=parsed.scheme=='rediss', storage_namespace=os.getenv('RAY_external_storage_namespace')) else None\""
+	} else {
+		pod.Spec.Containers[utils.RayContainerIndex].Args[0] += "sys.exit(1) if not cleanup_redis_storage(host=parsed.hostname, port=parsed.port, password=os.getenv('REDIS_PASSWORD', parsed.password), use_ssl=parsed.scheme=='rediss', storage_namespace=os.getenv('RAY_external_storage_namespace')) else None\""


We add the username=os.getenv('REDIS_USERNAME', parsed.username) positional argument only when the REDIS_USERNAME env is present because the cleanup_redis_storage function in old Ray doesn't support the argument.

Signed-off-by: Rueian <[email protected]>

kevin85421 · 2025-01-05T19:02:34Z

Would you mind separating this into several smaller PRs? Perhaps open a PR to add the fields in the CRD as the first step.

rueian · 2025-01-05T19:17:58Z

Would you mind separating this into several smaller PRs? Perhaps open a PR to add the fields in the CRD as the first step.

@kevin85421, no problem! Here is the first step which only contains the addition to the CRD. #2715

rueian · 2025-01-08T23:13:01Z

This will be split into even smaller PRs.

Moonquakes · 2025-01-09T01:08:05Z

Hi @rueian, I had previously inquired whether turning on GcsFaultTolerance was effective for Ray Cluster, i.e. whether the data corresponding to the Ray Dashboard could be recovered after the head node reboot, and the answer I got was no.

I would like to ask if this feature is designed to support this?

rueian · 2025-01-09T02:16:30Z

Hi @Moonquakes, The answer is still no. Ray only stores its cluster metadata in the Global Control Service (GCS); other data, such as metrics and logs, are not stored there. You will need to prepare additional external log storage (e.g., NFS) and metrics storage (e.g., Prometheus) for them if you want them to be recovered from failures.

Moonquakes · 2025-01-09T02:23:28Z

@rueian Thank you for your answer! However, GCS should also store task-related information. I don’t need to obtain logs and indicators. I just want to restore the original ray job and task information.

[RayCluster][Feature] add GcsFaultToleranceOptions to the RayCluster CRD

1252a6d

Signed-off-by: Rueian <[email protected]>

rueian force-pushed the gcs-ft-crd branch from 31cfe2a to 65bf9c9 Compare January 3, 2025 20:33

rueian changed the title ~~Gcs ft crd~~ [RayCluster][Feature] add GcsFaultToleranceOptions to the RayCluster CRD Jan 3, 2025

rueian added 2 commits January 4, 2025 05:06

[RayCluster][Feature] setup container envs by the GcsFaultToleranceOp…

6ac6c5a

…tions Signed-off-by: Rueian <[email protected]>

[RayCluster][Feature] support redis username in the GCS FT cleanup

c10db5c

Signed-off-by: Rueian <[email protected]>

rueian force-pushed the gcs-ft-crd branch from 65bf9c9 to c10db5c Compare January 3, 2025 21:06

rueian added 4 commits January 4, 2025 07:59

Merge remote-tracking branch 'origin/master' into gcs-ft-crd

03b8735

[RayCluster][Feature] add GcsFaultToleranceOptions to the helm CRD

ce630a6

Signed-off-by: Rueian <[email protected]>

[RayCluster][Feature] support redis username in the GCS FT cleanup

e3684fd

Signed-off-by: Rueian <[email protected]>

[RayCluster][Feature] modify rayStartParams with redis-username and r…

976a090

…edis-password Signed-off-by: Rueian <[email protected]>

rueian commented Jan 4, 2025

View reviewed changes

rueian force-pushed the gcs-ft-crd branch from 5a546fe to 8e3f506 Compare January 4, 2025 18:51

[RayCluster][Feature] add an e2e test for GcsFaultToleranceOptions

9465ae1

Signed-off-by: Rueian <[email protected]>

rueian force-pushed the gcs-ft-crd branch from 8e3f506 to 9465ae1 Compare January 4, 2025 18:52

rueian marked this pull request as ready for review January 4, 2025 18:58

rueian mentioned this pull request Jan 5, 2025

[RayCluster][Feature] add GcsFaultToleranceOptions to the RayCluster CRD [1/N] #2715

Merged

rueian marked this pull request as draft January 5, 2025 19:13

rueian mentioned this pull request Jan 7, 2025

[GCS FT] GCS FT misconfiguration #2694

Closed

2 tasks

Merge branch 'master' into gcs-ft-crd

614454a

rueian closed this Jan 8, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RayCluster][Feature] add GcsFaultToleranceOptions to the RayCluster CRD #2712

[RayCluster][Feature] add GcsFaultToleranceOptions to the RayCluster CRD #2712

rueian commented Jan 3, 2025 •

edited

Loading

rueian Jan 4, 2025

rueian Jan 4, 2025

rueian Jan 4, 2025

rueian Jan 4, 2025

rueian Jan 4, 2025 •

edited

Loading

kevin85421 commented Jan 5, 2025

rueian commented Jan 5, 2025

rueian commented Jan 8, 2025

Moonquakes commented Jan 9, 2025

rueian commented Jan 9, 2025

Moonquakes commented Jan 9, 2025

[RayCluster][Feature] add GcsFaultToleranceOptions to the RayCluster CRD #2712

[RayCluster][Feature] add GcsFaultToleranceOptions to the RayCluster CRD #2712

Conversation

rueian commented Jan 3, 2025 • edited Loading

Why are these changes needed?

Related issue number

Checks

rueian Jan 4, 2025

Choose a reason for hiding this comment

rueian Jan 4, 2025

Choose a reason for hiding this comment

rueian Jan 4, 2025

Choose a reason for hiding this comment

rueian Jan 4, 2025

Choose a reason for hiding this comment

rueian Jan 4, 2025 • edited Loading

Choose a reason for hiding this comment

kevin85421 commented Jan 5, 2025

rueian commented Jan 5, 2025

rueian commented Jan 8, 2025

Moonquakes commented Jan 9, 2025

rueian commented Jan 9, 2025

Moonquakes commented Jan 9, 2025

rueian commented Jan 3, 2025 •

edited

Loading

rueian Jan 4, 2025 •

edited

Loading