Telemetry: telemetry-summarize failing across RAPIDS: "failed to parse client cert pair"

[jameslamb]

Description

Across RAPIDS repos that have telemetry-summarize jobs in PR CI, I've been seeing failures like this for a few days:

....
Run ./shared-actions/telemetry-impls/sanity-checks
Run echo "OTEL_SERVICE_NAME is ${OTEL_SERVICE_NAME}"
OTEL_SERVICE_NAME is pr-cugraph
Run set -e
JOB_SPAN_ID pre-hash: "249365c21a0d1bfd546612f8c4af60d5-pr-cugraph"
STEP_SPAN_ID pre-hash: "249365c21a0d1bfd546612f8c4af60d5-pr-cugraph-"
JOB_SPAN_ID pre-hash: "249365c21a0d1bfd546612f8c4af60d5-pr-cugraph"
STEP_SPAN_ID pre-hash: "249365c21a0d1bfd546612f8c4af60d5-pr-cugraph-Set up job"
2024/12/06 23:26:45 failed to parse client cert pair: tls: failed to find any PEM data in certificate input

(example cugraph build)

This isn't technically blocking CI (as the jobs are optional), but it results in an ❌ next to CI in the UI. Can we fix this, to avoid forming the habit of ignoring that ❌ and to reduce the need to explain to folks that this job can be safely ignored?

Reproducible Example

Look at any recent PR CI job in cudf, cugraph, or rmm.

e.g. https://github.com/rapidsai/cudf/actions/workflows/pr.yaml

[msarahan]

Shared-workflows has a gate on an env var that skips telemetry:
rapidsai/shared-workflows#270

PRs to add this to repos that had telemetry set up already are here:

• gate telemetry dispatch calls on TELEMETRY_ENABLED env var cuml#6171
• gate telemetry dispatch calls on TELEMETRY_ENABLED env var cugraph#4816
• gate telemetry dispatch calls on TELEMETRY_ENABLED env var cudf#17551
• gate telemetry dispatch calls on TELEMETRY_ENABLED env var rmm#1752