Getting 403 forbidden error in analysisrun while integrating with datadog metrics query

[sv6375261073]

Hi All,

I am getting 403 error in analysisrun when new deployment version detects in rollouts. I verified the the same datadog query using python working fine, but analysisrun raising 403. Because of this deployment stucks and not proceeding further.

I am using argo-rollouts-2.34.1 version. what can be issue.

Secret:

apiVersion: v1
kind: Secret
metadata:
  name: datadog
  namespace: argo-rollouts
type: Opaque
stringData:
  address: ''
  api-key: ''
  app-key: ''

Analysis template:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  args:
  - name: service-name
  metrics:
  - name: error-rate
    interval: 5m
    successCondition: result <= 0.01
    failureLimit: 3
    provider:
      datadog:
        apiVersion: v2
        interval: 5m
        query: |
          sum:requests.error.rate{service:{{args.service-name}}}```
          
canary-config:
  
```yaml
canary:
  analysis:
    templates:
    - templateName: success-rate
    startingStep: 1
    args:
    - name: service-name
      value: canary-svc
    - name: cluster-name
      value: cluster

Error:

- consecutiveError: 5 error: 5 measurements: - finishedAt: '2024-02-21T18:46:20Z' message: >- received authentication error response code: 403 {"errors":["Forbidden"]}

Note: Analysisrun was run properly first time when, I had configured. But later started getting this issue but couldn't get fix.