[opentelemetry-collector] Make target allocator deployable via opentelemetry-collector helm chart

[zoranzaric-cpx]

As suggested by @jaronoff97 via Slack here's a feature request to make the target allocator deployable via the collector helm chart.

My goal would be to have collector metric scrape_configs generated from ServiceMonitor CRDs.

[TylerHelmuth]

@zoranzaric-cpx is this issue related to this request: #310

[zoranzaric-cpx]

@TylerHelmuth I'm reading #310 as a feature for the operator. I'm requesting making the target allocator usable without the operator just with the collector helm chart.

[TylerHelmuth]

I'm not super familiar with the otel-allocator housed in the operator repo. Its goal is to load balance Prometheus endpoints that the collector's Prometheus receiver would scrape? Can it be used independently from the operator?

[zoranzaric-cpx]

I'm not sure, @jaronoff97 suggested I create an issue on slack: https://cloud-native.slack.com/archives/C01N6P7KR6W/p1660910465791709

[TylerHelmuth]

That thread reads to me like the target allocator needs to be managed via the operator. There is an operator chart, that would probably be best place to enable this functionality, unless I'm mis-understanding.

[jaronoff97]

I believe the target allocator should be possible to be used separate from the operator, given there is a public container published for the image. The operator helps manage some of the details necessary for the configuration of the collector and target-allocator, which should be possible for the standalone collector helm chart. @TylerHelmuth you can assign this issue to me to investigate.

[TylerHelmuth]

@jaronoff97 sounds good to me, interested to see the results.

[zoranzaric-cpx]

@jaronoff97 if there's legwork to do, I'll be happy to do it with some mentoring.

[jaronoff97]

@zoranzaric-cpx thanks for offering! It's mostly cut and pasting which i've now done. Would appreciate your review on the PR once it's ready, however :)

[husseinraoouf]

I hope this pr get more attention

@jaronoff97 do you need any help, I can continue where you left if you could point me to the right direction

[husseinraoouf]

I also think the chart should include PodMontior and ServiceMonitor CRD and allow the target allocator service account to watch them if prometheusCR is enabled in the target allocator config

[jaronoff97]

@husseinraoouf I have a working branch here that has what I've done so far. Most of my focus has been on improving the target allocator in the operator repository and helm support proved to be more difficult than I thought. Ultimately what needs to be done for my branch is implement the logic to do all of the substitution the operator does when instantiating the configmap. This task is a bit annoying to do in helm, not impossible, just very tedious. If you'd like to take a go, that would be of great help, if not, for now I would recommend using the operator and target allocator to accomplish what you'd like.

[povilasv]

FYI We use Target Allocator without Operator :) It's rather simple you can pass url of Target Allocator to Prometheus receiver and it will convert ServiceMonitors / PodMonitors to scrape configs. Example:

      prometheus:
        config:
          scrape_configs:
          - job_name: opentelemetry-collector
            scrape_interval: 30s
            static_configs:
            - targets:
              - ${env:MY_POD_IP}:8888
        target_allocator:
          collector_id: ${env:MY_POD_NAME}
          endpoint: http://opentelemetry-targetallocator
          interval: 30s

I'm not sure whether this should be bundled with OpenTelemetry Collector chart?

We could create a preset that allows you to automatically scrape ServiceMonitor / PodMonitor 🤔

On the other hand separate helm chart for TA also makes sense, then users get the freedom to do whatever they want and can bundle thru subcharts.

[jaronoff97]

@nicolastakashi going to carry your comment over here... I could see that being valuable, especially if a user is okay with not getting the convenience that you would get for the TA via the operator. I think if we were to bundle this in an existing chart, having a solo target allocator section in kube-stack chart may make sense, otherwise yes a new chart would be the way to go. Let's maybe chat about the tradeoffs in slack to sort through it and come back here with some consensus.