Proposal: Enhance echo component for better insights

[bentonam]

Background

The loki.echo component is useful, however, if you're using a bunch of modules as an example. You don't have a way to debug or get insights into that very easily, without downloading all of those modules and changing them. For example take say that https://github.com/grafana/agent-modules/blob/main/modules/kubernetes/logs/all.river is my config. To add a loki.echo and filter the results to a specific pod/service while still sending my logs to loki. I would have to do something like:

module.git "label_keep" {
  repository = argument.git_repo.value
  revision = argument.git_rev.value
  pull_frequency = argument.git_pull_freq.value
  path = "modules/kubernetes/logs/labels/keep-labels.river"

  arguments {
    forward_to = loki.write.destination.receiver
    keep_labels = argument.keep_labels.value
  }
}

loki.process "split" {
  forward_to = [loki.write.destination.receiver, loki.process.testing.receiver]
}

loki.process "testing" {
  forward_to = [loki.echo.testing.receiver]
  // stages to only keep matching / relevant messages
}

loki.echo "testing" {}

loki.write "destination" {
  endpoint {
    url = argument.loki_url.value

    basic_auth {
        username = argument.loki_username.value
        password = argument.loki_password.value
    }
  }

  external_labels = {
    "cluster" = argument.label_cluster.value,
    "env" = argument.label_env.value,
    "region" = argument.label_region.value,
  }
}

While I would see these in the console output from loki.echo filtered to the messages I want, I would still need to go to the console, view the logs, etc.

Proposal

Add a new optional argument to loki.echo of selector that can execute a LogQL selector just like the component loki.process -> stage.match which will filter the logs that get echoed.

Additionally, from the UI it is not very easy to get insights into what is going on within the loki.process, where at least for prometheus.scrape components, you can see the arguments that came in from service discovery but also each target that was scraped, the set of labels associated to that scrape from service discovery, when it was last successful, etc.

To further enhance the UI experience for logs, leverage web sockets, so that anytime a message is sent to loki.echo it will write to both the console and publish that same message to the web socket, so that in real-time from the browser, I can see the messages coming in. This would allow for easier debugging and insights, viewing of what labels are associated with the log message, etc.

This would enhance metrics as well by introducing a prometheus.echo that allows you to see the exact metrics that are being written with the combined service discovery labels plus the scraped labels and be able to filter the results down based on a PromQL label selector of what is echoed directly from the console or the UI.

[ptodev]

Components such as loki.process often do very complex transformations. I agree that we should provide a way for users to easily see how those processes transform the telemetry. However, I wish we could do it via telemetry signals we already support in the Agent (logs/metrics/traces). In this case, would it not be appropriate to:

• Make sure we have enough debug logging in loki.process
• Create a new feature in the Agent so that debug logging can be enabled on a per-component basis, instead of being global. E.g. it may only be enabled for loki.process.default. That way users will be able to see how a certain component works without even adding new components such as an echo one. This pattern could also be applied to non-loki components as well.

[thampiotr]

We have built a new feature called live debugging, which will cover this use-case very soon. Some of it is already available in our main branch. Tracking issue: #785 - setting this to Likely Decline, as we believe it will be covered by the new live debugging feature.

[rfratto]

cc @bentonam Do you think this adds anything beyond what #785 provides?

cc @wildum

[Musatra]

How to do that ?