Fix telemetry api subscription 403 forbidden error

[jerrytfleung]

Problem

We see the following telemetry API subscription error when we use telemetry API receiver and setup extensions in collector lambda layer.

e.g. config.yaml

service:
  # Extensions specified below are going to be loaded by the service in the
  # order given below, and shutdown on reverse order.
  extensions: [solarwindsapmsettingsextension]

error message

{
    "level": "info",
    "ts": 1721258635.0392153,
    "msg": "done",
    "error": "cannot start pipelines: request to http://127.0.0.1:9001/2022-07-01/telemetry failed: 403[403 Forbidden] {\"errorMessage\":\"Telemetry API subscription is closed already\",\"errorType\":\"Telemetry.SubscriptionClosed\"}\n; Post \"http://127.0.0.1:9001/2020-01-01/extension/init/error\": net/http: invalid header field value for \"Lambda-Extension-Function-Error-Type\"",
    "errorCauses": [
        {
            "error": "cannot start pipelines: request to http://127.0.0.1:9001/2022-07-01/telemetry failed: 403[403 Forbidden] {\"errorMessage\":\"Telemetry API subscription is closed already\",\"errorType\":\"Telemetry.SubscriptionClosed\"}\n"
        },
        {
            "error": "Post \"http://127.0.0.1:9001/2020-01-01/extension/init/error\": net/http: invalid header field value for \"Lambda-Extension-Function-Error-Type\""
        }
    ]
}

Investigation

1. Telemetry API subscription is closed already means the subscription was sent after EXTENSION INIT phase and thus the subscription is closed.
2. From here, the processEvents function is called from a go routine and it is before the creation of collector and its components. Calling processEvents function in EXTENSION INIT phase means signalling AWS Lambda service that this lambda layer is initialized. So there is a race condition between telemetry API subscription and processEvents signalling the extension is initialized.

Solution

With the consideration of telemetry API receiver, we need to make sure the collector is up and running before signalling AWS Lambda service the lambda layer is initialized.
This PR is to fix this problem.

[tylerbenson]

Sounds reasonable to me.