Grafana doesn't detect Pyroscope correctly

[nm-bingushim]

Hello,

Environment

• Pyroscope : 1.0.0 (installed using Helm)
• Grafana : 10.1.4

Problem Details

I installed Pyroscope using helm chart and It works okay both accepting Java Profile data and default dashboard.

When I try to add the Pyroscope cluster as a Datasource to our Grafana, I got warning as follow.

If i click 'Save & test' button, I got this error message.

When I change backend type to Phlare, the warn message disappeared.

Here is the values.yaml I used to install pyroscope.
The Grafana version is 10.1.4, which is currently lates version.)

# Default values for pyroscope.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

pyroscope:
  replicaCount: 3

  image:
    repository: grafana/pyroscope
    pullPolicy: IfNotPresent
    # Overrides the image tag whose default is the chart appVersion.
    tag: "1.0.0"

  extraArgs:
    log.level: info

    # The following environment variables are set by the Helm chart.
  # JAEGER_AGENT_HOST: jaeger-agent.jaeger.svc.cluster.local.

  # -- Environment variables from secrets or configmaps to add to the pods
  extraEnvFrom: []

  imagePullSecrets: []
  nameOverride: ""
  fullnameOverride: ""

  serviceAccount:
    # Specifies whether a service account should be created
    create: true
    # Annotations to add to the service account
    annotations: {}
    # The name of the service account to use.
    # If not set and create is true, a name is generated using the fullname template
    name: ""

  podAnnotations:
    # Scrapes itself see https://grafana.com/docs/phlare/latest/operators-guide/deploy-kubernetes/#optional-scrape-your-own-workloads-profiles
    profiles.grafana.com/memory.scrape: "true"
    profiles.grafana.com/memory.port_name: http2
    profiles.grafana.com/cpu.scrape: "true"
    profiles.grafana.com/cpu.port_name: http2
    profiles.grafana.com/goroutine.scrape: "true"
    profiles.grafana.com/goroutine.port_name: http2
    # profiles.grafana.com/block.scrape: "true"
    # profiles.grafana.com/mutex.scrape: "true"

  podSecurityContext:
    fsGroup: 10001
    runAsUser: 10001
    runAsNonRoot: true

  securityContext:
    {}
    # capabilities:
    #   drop:
    #   - ALL
    # readOnlyRootFilesystem: true
    # runAsNonRoot: true
  # runAsUser: 1000

  service:
    type: ClusterIP
    port: 4040
    port_name: http2

  memberlist:
    port: 7946
    port_name: memberlist

  resources:
    {}
    # We usually recommend not to specify default resources and to leave this as a conscious
    # choice for the user. This also increases chances charts run on environments with little
    # resources, such as Minikube. If you do want to specify resources, uncomment the following
    # lines, adjust them as necessary, and remove the curly braces after 'resources'.
    #
    # Note that if memory consumption is higher than you would like, you can decrease the interval
    # that profiles are written into blocks by setting `pyroscopedb.max-block-duration` in the `extraArgs`
    # stanza. By default, it is set to 3h - override it, for example, as below:
    # ```
    # extraArgs:
    #   pyroscopedb.max-block-duration: 30m
    # ```
    #
    # limits:
    #   cpu: 100m
    #   memory: 128Mi
    # requests:
    #   cpu: 100m
  #   memory: 128Mi

  nodeSelector: {}

  extraEnvVars:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: "/etc/gcs_secrets/gcs-serviceaccount.json"
  extraVolumeMounts:
    - name: gcs-secrets
      mountPath: "/etc/gcs_secrets"
  extraVolumes:
    - name: gcs-secrets
      secret:
        secretName: heimdall-gcs-serviceaccount

  ## ref: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
  ## If you set enabled as "True", you need :
  ## - create a pv which above 10Gi and has same namespace with phlare
  ## - keep storageClassName same with below setting
  persistence:
    enabled: true
    storageClassName: fast
    accessModes:
      - ReadWriteOnce
    size: 10Gi
    annotations: {}
    # selector:
    #   matchLabels:
    #     app.kubernetes.io/name: phlare
    # subPath: ""
    # existingClaim:

  tolerations:
    - key: "spec"
      operator: "Equal"
      value: "pyroscope"
      effect: "NoSchedule"

  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
          matchExpressions:
            - key: app.kubernetes.io/component
              operator: In
              values:
                - pyroscope
          topologyKey: "kubernetes.io/hostname"
  # run specific components separately
  components: {}

  # -- Allows to override Phlare's configuration using structured format.
  structuredConfig: {}

  # -- Contains Phlare's configuration as a string.
  # @default -- The config depends on other values been set, details can be found in [`values.yaml`](./values.yaml)
  config: |
    limits:
      max_query_lookback: 30d
      max_query_length: 7d
    storage:
      backend: gcs
      storage_prefix: /var/pyroscope
      gcs:
        bucket_name: heimdall-pyroscope-dev1

  # -- Allows to add tenant specific overrides to the default limit configuration.
  tenantOverrides:
    {}
    # "foo":
    #   ingestion_rate_mb: 1
  #   ingestion_burst_size_mb: 2
  # -- Grafana Agent Configuration.

# -------------------------------------
# Configuration for `grafana-agent` child chart
# -------------------------------------
agent:
  enabled: false


# -------------------------------------
# Configuration for `minio` child chart
# -------------------------------------
minio:
  enabled: false

ingress:
  enabled: false
  # hosts:
  #   - localhost

I'm posting this question here, since I think i missed something when I install Pyroscope.

[cyriltovena]

@aocenas Do you remember when we do send this error message ?

[aocenas]

So this was in an older version of Grafana. The problem was that at that point we had old Phlare API and Pyroscope API so we had to know what kind of client to use. After that the new version of Pyroscope adopted the Phlare API structure so it created this weird thing where new Pyroscope backend would be detected as Phlare. We then removed the config as we effectivelly deprecated Phlare. So this autodetection is weird because of the naming but technically correct if you use older Grafana + new Pyroscope.