Rework CI Visibility spans batching

[nikita-tkachenko-datadog]

Requirements for Contributing to this repository

• Fill out the template below. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
• The pull request must only fix one issue at the time.
• The pull request must update the test suite to demonstrate the changed functionality.
• After you create the pull request, all status checks must be pass before a maintainer reviews your contribution. For more details, please see CONTRIBUTING.

What does this PR do?

This PR adds logic that allows batching pipeline/job spans that the plugin submits: the logic that monitors jobs and pipelines calls the new TraceWriter to add a span to the current batch. The writer runs a background thread that gathers submitted spans into batches and submits them.

Each batch includes up to 100 spans (configurable via env vars), and the thread polls the queue at least every 10 seconds (configurable via env vars) or more frequently if the queue has more spans than fit into a single batch.

The motivation for adding this is:

• Preparing for future changes where pipeline steps/stages will be submitted as soon as they finish, rather than at the end of the pipeline.
• Optimising things by batching different jobs/pipelines together into a single request.

There was a batching mechanism that existed before the changes (NonBlockingHttpClient), however it did not address the two points above: different jobs could not be batched together, and all of the spans had to be submitted in one call (which mandated waiting until the end of the pipeline to accumulate them).

Description of the Change

Alternate Designs

Possible Drawbacks

Verification Process

This does not change the plugin behaviour, so existing tests were used to verify there were no regressions.
In addition, manual tests were executed in a dockerized Jenkins instance, covering the following:

• Freestyle build trace submitted via webhooks
• Freestyle build trace submitted via EVP proxy
• Freestyle build trace submitted via APM track
• Pipeline trace submitted via webhooks
• Pipeline trace submitted via EVP proxy
• Pipeline trace submitted via APM track

Additional Notes

Release Notes

Review checklist (to be filled by reviewers)

• Feature or bug fix MUST have appropriate tests (unit, integration, etc...)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have one changelog/ label attached. If applicable it should have the backward-incompatible label attached.
• PR should not have do-not-merge/ label attached.
• If Applicable, issue must have kind/ and severity/ labels attached at least.

[drodriguezhdez]

I didn't finish the review yet, but I've dropped some comments in the TracerWriter part.