loki ingress endpoint doesn't work

[simondeziel]

Bug Description

After deploying the COS Lite bundle (https://github.com/canonical/cos-lite-bundle), the Grafana dashboard datasources section looks like this:

The Alertmanager and Prometheus datasources successfully pass the Grafana checks but the Loki one fails due to receiving a 404 error.

To Reproduce

1. Follow the "Deploy the COS Lite bundle with overlays" section of https://charmhub.io/topics/canonical-observability-stack/install/microk8s
2. juju deploy cos-lite --trust --channel=edge --overlay ./offers-overlay.yaml --overlay ./storage-small-overlay.yaml

Environment

1. juju status

ubuntu@microk8s:~$ juju status --relations
Model  Controller  Cloud/Region         Version  SLA          Timestamp
cos    overlord    k8s-cloud/localhost  2.9.34   unsupported  15:41:05Z

App           Version  Status  Scale  Charm             Channel  Rev  Address         Exposed  Message
alertmanager  0.23.0   active      1  alertmanager-k8s  edge      32  10.152.183.115  no       
grafana       8.2.6    active      1  grafana-k8s       edge      44  10.152.183.212  no       
loki          2.4.1    active      1  loki-k8s          edge      45  10.152.183.113  no       
prometheus    2.33.5   active      1  prometheus-k8s    edge      71  10.152.183.116  no       
traefik                active      1  traefik-k8s       edge      88  10.76.66.104    no       

Unit             Workload  Agent  Address       Ports  Message
alertmanager/0*  active    idle   10.1.128.202         
grafana/0*       active    idle   10.1.128.251         
loki/0*          active    idle   10.1.128.252         
prometheus/0*    active    idle   10.1.128.200         
traefik/0*       active    idle   10.1.128.249         

Offer                            Application   Charm             Rev  Connected  Endpoint              Interface                Role
alertmanager-karma-dashboard     alertmanager  alertmanager-k8s  32   0/0        karma-dashboard       karma_dashboard          provider
grafana-dashboards               grafana       grafana-k8s       44   0/0        grafana-dashboard     grafana_dashboard        requirer
loki-logging                     loki          loki-k8s          45   0/0        logging               loki_push_api            provider
prometheus-receive-remote-write  prometheus    prometheus-k8s    71   0/0        receive-remote-write  prometheus_remote_write  provider
prometheus-scrape                prometheus    prometheus-k8s    71   0/0        metrics-endpoint      prometheus_scrape        requirer

Relation provider                   Requirer                     Interface              Type     Message
alertmanager:alerting               loki:alertmanager            alertmanager_dispatch  regular  
alertmanager:alerting               prometheus:alertmanager      alertmanager_dispatch  regular  
alertmanager:grafana-dashboard      grafana:grafana-dashboard    grafana_dashboard      regular  
alertmanager:grafana-source         grafana:grafana-source       grafana_datasource     regular  
alertmanager:replicas               alertmanager:replicas        alertmanager_replica   peer     
alertmanager:self-metrics-endpoint  prometheus:metrics-endpoint  prometheus_scrape      regular  
grafana:grafana                     grafana:grafana              grafana_peers          peer     
loki:grafana-dashboard              grafana:grafana-dashboard    grafana_dashboard      regular  
loki:grafana-source                 grafana:grafana-source       grafana_datasource     regular  
loki:metrics-endpoint               prometheus:metrics-endpoint  prometheus_scrape      regular  
prometheus:grafana-dashboard        grafana:grafana-dashboard    grafana_dashboard      regular  
prometheus:grafana-source           grafana:grafana-source       grafana_datasource     regular  
prometheus:prometheus-peers         prometheus:prometheus-peers  prometheus_peers       peer     
traefik:ingress-per-unit            loki:ingress                 ingress_per_unit       regular  
traefik:ingress-per-unit            prometheus:ingress           ingress_per_unit       regular  
traefik:metrics-endpoint            prometheus:metrics-endpoint  prometheus_scrape      regular

2. snap list

Name        Version      Rev    Tracking       Publisher   Notes
charmcraft  2.0.0        1033   latest/stable  canonical✓  classic
core18      20220831     2566   latest/stable  canonical✓  base
core20      20220826     1623   latest/stable  canonical✓  base
juju        2.9.34       20510  2.9/stable     canonical✓  classic
lxd         5.6-794016a  23680  latest/stable  canonical✓  -
microk8s    v1.25.0      3883   1.25/stable    canonical✓  classic
snapd       2.57.2       17029  latest/stable  canonical✓  snapd

Relevant log output

`juju debug-log --replay`: https://termbin.com/wndl

Additional context

No response

[simskij]

I can confirm this is the case. It seems like the webroot isn't properly set as removing the relation between loki and traefik and refreshing the source turns it green. Adding it back again and refreshing the datasources, however, it continues to work, so this might very well be connected to canonical/traefik-k8s-operator#78, as that usually manifests by deploying it as a bundle.

[dstathis]

When I deploy the bundle this does not happen for me. Can anyone confirm that this bug still exists?

[dstathis]

When I deploy the bundle this does not happen for me. Can anyone confirm that this bug still exists?

@simskij

[simskij]

I can't reproduce it either, so seems like the fixes to traefik solved it.