How to configure ingress-nginx metrics to show in Prometheus?

[sonamsamdupkhangsar]

I have followed the starter kit using the Nginx Ingress controller instead of the Ambassador Ingress. I want to expose the Nginx Ingress metrics in Prometheus so I can see my app logs in Loki. This is what I have done to expose Nginx Ingress metrics:

NGINX_CHART_VERSION="4.0.6";
helm upgrade ingress-nginx ingress-nginx/ingress-nginx --version "$NGINX_CHART_VERSION" --namespace ingress-nginx -f "03-setup-ingress-controller/assets/manifests/nginx-values-v${NGINX_CHART_VERSION}.yaml"  --set controller.metrics.enabled=true  --set-string controller.podAnnotations."prometheus\.io/scrape"="true"  --set-string controller.podAnnotations."prometheus\.io/port"="10254"

I can see the Nginx Ingress metrics in this command:

kubectl get svc -n ingress-nginx                                    
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.245.57.3      <REDACTED>   80:31150/TCP,443:31740/TCP   9d
ingress-nginx-controller-admission   ClusterIP      10.245.186.61    <none>            443/TCP                      9d
ingress-nginx-controller-metrics     ClusterIP      10.245.240.243   <none>            10254/TCP                    20h

I port-forwarded the ingress-nginx-controller-metrics and to check the metrics:

kubectl port-forward svc/ingress-nginx-controller-metrics 10254:10254 -n ingress-nginx
Forwarding from 127.0.0.1:10254 -> 10254

Check the Ingress metrics using curl:

curl -s http://localhost:10254/metrics | more
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.0449e-05
go_gc_duration_seconds{quantile="0.25"} 7.6121e-05
go_gc_duration_seconds{quantile="0.5"} 8.6591e-05
go_gc_duration_seconds{quantile="0.75"} 0.000110908
...

So kind of like how it is done for the Ambassador Ingress stack here I put my changes in the additionalServiceMonitors section as follows:

additionalServiceMonitors:
  - name: "ingress-nginx-monitor"
    selector:
      matchLabels:
        service: "ingress-nginx-controller-metrics"   
    namespaceSelector:
      matchNames:
        - ingress-nginx
    endpoints:
      - port: "ingress-nginx-controller-metrics"
        path: /metrics
        scheme: http

I applied this change with Helm upgrade :

 HELM_CHART_VERSION="27.2.1";
helm upgrade kube-prom-stack prometheus-community/kube-prometheus-stack --version "${HELM_CHART_VERSION}"  --namespace monitoring  -f "04-setup-prometheus-stack/assets/manifests/prom-stack-values-v${HELM_CHART_VERSION}.yaml"

(I chose 27.2.1 version for kube-prometheus-stack as there was a updated one)

I then port-forward my Prometheus to check on the Status:
kubectl port-forward svc/kube-prom-stack-kube-prome-prometheus 9090:9090 -n monitoring

I then, navigate to Status -> Targets page but I don't see my Nginx Ingress there kind of like how it is mentioned for the Ambassador Ingress version.

Any ideas on how to get the Nginx Ingress metrics in Prometheus?

[vladciobancai]

Hi @sonamsamdupkhangsar to enable the Nginx Ingress Controller metrics in prometheus please use the following values

  additionalServiceMonitors:
    - name: "ingress-nginx-monitor"
      selector:
        matchLabels:
          app.kubernetes.io/name: ingress-nginx
      namespaceSelector:
        matchNames:
          - ingress-nginx
      endpoints:
        - port: "metrics"

Run the helm upgrade command for kube-prometheus-stack, at the end the metrics will be available (below is an example from my test)

[sonamsamdupkhangsar]

Thank you @vladciobancai, this worked.