How to debug scraping

[eviscares]

Hey,
I'm coming to you with a problem that keeps bugging me, namely the issue of debugging what the agent is doing (or not doing) when used to scrape metrics to forward to a prometheus.
My particular issue is that I'm running a pod that has all the right annotations (at least as per my understanding), but nothing gets scraped.
When I curl the path I'm seeing prometheus style metrics. And debugging this feels like a very extensive game of trial and error, so I'm wondering if there is a better way to get some info out of the agent.

For reference, here's the annotations:

prometheus.io/path: /actuator/metrics
prometheus.io/port: '58180'
prometheus.io/scrape: 'true'

and here's the job config:

- job_name: "kubernetes-pods"
              bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
              kubernetes_sd_configs:
                - role: pod
              relabel_configs:
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                  action: keep
                  regex: true
                - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
                  action: replace
                  target_label: __metrics_path__
                  regex: (.+)
                - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
                  action: replace
                  regex: ([^:]+)(?::\d+)?;(\d+)
                  replacement: $1:$2
                  target_label: __address__
                # Expose Pod labels as metric labels
                - action: labelmap
                  regex: __meta_kubernetes_pod_label_(.+)
                # Expose Pod namespace as metric namespace label
                - source_labels: [ __meta_kubernetes_namespace ]
                  action: replace
                  target_label: namespace
                # Expose Pod name as metric name label
                - source_labels: [ __meta_kubernetes_pod_name ]
                  action: replace
                  target_label: pod
                  ```