Add optional gateway collector for the opentelemetry-kube-stack

[rogercoll]

The current kube-stack Chart does not provide any specific collector to be used as a data gateway. The alternative is to configure the daemonset collectors to be used as gateways for OTLP data. Although this works fine, adding an optional gateway collector would include additional benefits:

• Separation of concerns: Decouples data ingestion from processing. It allows the Daemonset and Standalone collectors to focus on their specific tasks (e.g., collecting infrastructure metrics, logs, and events) without having to handle the complexity of processing and routing data to different backends. The daemonset would be seen as an agent.
• Isolation and simplified networking configuration.
  • Network issues with backends would be easier to troubleshoot.
• Centralized data aggregation/enrichment
• Easier scale mechanism: Instead of being tied to pod per node doing data aggregations (daemon), the gateway would be a deployment that can scale independently of the number of nodes (or taint the gateway collector specific hardware pods).
• Load distribution: It would handle all the OTLP load currently done by the daemonset collectors. An auto-scaling replicaset could be used based on the workload.

Expected result

This chart installs the OpenTelemetry Operator and two collector pools with the following features:

    Daemonset collector
        Kubernetes infrastructure metrics
        Applications logs
        OTLP trace receiver (Moved to the Gateway collector if enabled)
        Kubernetes resource enrichment
    Standalone collector
        Kubernetes events
        Cluster metrics
    (Optional) Gateway collector
        OTLP trace/metrics/logs receiver
        OTLP trace/metrics/logs exporter