feat(stackable-telemetry): Improve stackable-telemetry to make it easier to build the TraceGuard

[NickLarsenNZ]

Description

Part of stackabletech/issues#639.

Problem

It is cumbersome to configured the various subscribers to be enabled at runtime. Using @soenkeliebau's code example:

// Instead of building a TraceGuard, a mutable builer needs to be assigned so that...
let mut builder = Tracing::builder()
    .service_name("whoyougonnacall")
    .with_console_output("WYGC_CONSOLE", LevelFilter::INFO);

// ... builder methods can be called conditionally:
if enable_trace_exporter().context(ParseConfigSnafu)? {
    builder = builder.with_otlp_trace_exporter("TEST_OTLP_TRACE", LevelFilter::TRACE);
}
if enable_log_exporter().context(ParseConfigSnafu)? {
    builder = builder.with_otlp_log_exporter("TEST_OTLP_LOG", LevelFilter::TRACE);
}

// Then finally the trace TraceGuard is returned.
let _tracing_guard = builder.build().init().context(InitializeTelemetrySnafu)?;

The purpose of the builder is to allow overriding of the the defaults, however in the current state, the function parameters for the subscribers (console and otlp exporters) break that, and it will only get messier as config options are added.

Solution

Add common and subscriber specific configuration through a Settings builder (suggested by @Techassi). The previous example would then look like:

Caution

This is work in progress, so it might change, but the essence remains the same.
Todo: update this before merging.

let _trace_guard = Tracing::builder()
    .service_name("whoyougonnacall")
    .with_console_output(
        Settings::builder()
            .env_var("WYGC_CONSOLE")
            .default_level(LevelFilter::INFO)
            .enabled(true)
            // 👇 this part is new, and has no analogue in the example from the problem statement above.
            .console_log_settings_builder()
            .log_format(Format::Plain)
            // ☝️
            .build(),
    )
    .with_otlp_trace_exporter(
        Settings::builder()
            .env_var("TEST_OTLP_TRACE")
            .default_level(LevelFilter::TRACE)
            .enabled(enable_trace_exporter().context(ParseConfigSnafu)?)
            .build(),
    )
    .with_otlp_log_exporter(
        Settings::builder()
            .env_var("TEST_OTLP_LOG")
            .default_level(LevelFilter::TRACE)
            .enabled(enable_log_exporter().context(ParseConfigSnafu)?)
            .build(),
    )
    .build();

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

Beta Give feedback

1. Changes are OpenShift compatible
2. CRD changes approved
3. Integration tests passed (for non trivial changes)

Reviewer

Beta Give feedback

1. Code contains useful comments
2. (Integration-)Test cases added
3. Documentation added or updated
4. Changelog updated
5. Cargo.toml only contains references to git tags (not specific commits or branches)

[soenkeliebau]

I like it!

[NickLarsenNZ]

I like it!

Thanks.

It is basically done, there is one thing left to refactor (and enable console log formats), but before merging I will document the pub items (and enable warn on missing docs).