data-prepper/examples/jaeger-hotrod example NOT working as expected.

[contractormarketplace]

We are testing out the example jaeger-hotrod.

We are following the example on https://opendistro.github.io/for-elasticsearch-docs/docs/trace/get-started/

In the directory data-prepper/examples/jaeger-hotrod
we execute the command
sudo docker-compose -f docker-compose.yml up --build -d --force-recreate --remove-orphans

http://localhost:8080/ brings up the sample application

Jaeger UI is not coming up at http://localhost:16686/ - we don't see this port configured in the docker-compose.yml

The following curl command does not work, seems like the indexes are not set up in elastic search.
curl -X GET -u 'admin:admin' -k 'https://localhost:9200/otel-v1-apm-span-000001/_search?pretty&size=1'
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index [otel-v1-apm-span-000001]",
"resource.type" : "index_or_alias",
"resource.id" : "otel-v1-apm-span-000001",
"index_uuid" : "na",
"index" : "otel-v1-apm-span-000001"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index [otel-v1-apm-span-000001]",
"resource.type" : "index_or_alias",
"resource.id" : "otel-v1-apm-span-000001",
"index_uuid" : "na",
"index" : "otel-v1-apm-span-000001"
},
"status" : 404
}

We go to the following address and then navigate to Trace Analytics :
http://localhost:5601

We see the following error, there are no traces listed.
Indices required
The indices required for trace analytics (otel-v1-apm-span-* and otel-v1-apm-service-map) do not exist or you do not have permission to access them.

we do a sudo docker logs --follow otel-collector, we see some errors in the collector
2021-05-05T05:34:44.541Z info exporterhelper/queued_retry.go:314 Exporting failed. Will retry the request after interval. {"kind": "exporter", "name": "otlp/2", "error": "failed to push trace data via OTLP exporter: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup data-prepper on 127.0.0.11:53: server misbehaving"", "interval": "9.094969591s"}

Can you please clarify and help with understanding this example provided.

[chenqi0805]

@contractormarketplace Thank you for the interest in trying out data-prepper. Regarding the bug, it looks like your data-prepper container failed to start. But we have not yet been able to reproduce it through opendistroforelasticsearch-data-prepper-0.8.0-beta-linux-x64/examples/jaeger-hotrod or the current main. For further diagnostics, could you please share the docker logs on data-prepper container?

Jaeger UI is not coming up at http://localhost:16686/ - we don't see this port configured in the docker-compose.yml

No, we did not include Jaeger frontend in the docker-compose.

[contractormarketplace]

Hello @chenqi0805 thank you for your guidance. As you pointed out, there are problems with dataprepper which prevent it from starting up.

I fixed the first issue in the docker-compose.yml file , but other issues show up.

• ../data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml

Attaching the log file for you.
dataPrepper.log

[contractormarketplace]

Also, I could NOT find the branch you are referring to
https://github.com/opendistro-for-elasticsearch/data-prepper/branches

Is this branch available?
opendistroforelasticsearch-data-prepper-0.8.0-beta-linux-x64/examples/jaeger-hotrod

[chenqi0805]

@contractormarketplace Thanks for sharing the data-prepper logs. From the logs I can see that it failed to find the pipeline configuration file. Not sure which version of docker-compose file you are using. Notice that from 0.8.0+, we have two configuration files:

data-prepper/examples/jaeger-hotrod/docker-compose.yml

Lines 8 to 9 in 63034a6

	- ../trace_analytics_no_ssl.yml:/usr/share/data-prepper/pipelines.yaml
	- ../data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml

, one for pipelines and the other for data-prepper server running at 4900. I would recommend trying with the latest docker-compose file (in the current main branch or release tag v0.8.x) and see if the issue persists.

[chenqi0805]

Is this branch available?
opendistroforelasticsearch-data-prepper-0.8.0-beta-linux-x64/examples/jaeger-hotrod

Here I am referring to examples/jaeger-hotrod in Version 0.8.0-beta