Jaeger is an opensource distributed tracing system developed by Uber, it is mainly used for micro service scenarios. It can be used to analyze the invocation process for multiple services, display the method call trace and the method call relations. It is a useful tool for diagnosing performance problems and analyzing system failures.
Jaeger on Aliyun Log Service is a distributed tracing system based on Jaeger which supports persist data into Aliyun Log Service. What's more you can retrieve them from log service through jaeger-query and display them on Jaeger UI.
Jaeger clients are language specific implementations of the OpenTracing API. They can be used to instrument applications for distributed tracing either manually or with a variety of existing open source frameworks, such as Flask, Dropwizard, gRPC, and many more, that are already integrated with OpenTracing.
A network daemon that listens for spans sent over UDP, which it batches and sends to the collector. It is designed to be deployed to all hosts as an infrastructure component. The agent abstracts the routing and discovery of the collectors away from the client.
The collector receives traces from Jaeger agents and runs them through a processing pipeline. The storage is a pluggable component. Jaeger on Aliyun Log Service supports use Aliyun Log Service as the backend storage.
Query is a service that retrieves traces from storage and hosts a UI to display them.
The jaeger-collector will persist the received data to the log service. The jaeger-query will retrieve data from the log service.
Jaeger provides docker images that allows you to run various components in a convenient way. However, if you can't use docker in your environment, you can also build binary files that can run on the corresponding platform based on the source code directly or use the release packages.
To get started, make sure you clone the Git repository into the correct location github.com/jaegertracing/jaeger relative to $GOPATH:
mkdir -p $GOPATH/src/github.com/jaegertracing
cd $GOPATH/src/github.com/jaegertracing
git clone https://github.com/aliyun/aliyun-log-jaeger.git jaeger
cd jaeger
Then install dependencies:
git submodule update --init --recursive
make install
Please use the following commands to build the components that can run on the corresponding platform.
make build-all-linux
make build-all-windows
make build-all-darwin
Please configure the log service according to the following steps.
- Login on Aliyun Log Service Web Console.
- Create project, logstore for storing span.
- Create indexes for the following fields.
| Field Name | Type | Token |
|---|---|---|
| traceID | text | N/A |
| spanID | text | N/A |
| process.serviceName | text | N/A |
| operationName | text | N/A |
| startTime | long | N/A |
| duration | long | N/A |
Note: if you want to use tags as condition to find traces, you should alse create indexes for the tag fields. For example, the application generate the following tags http.method, http.status_code and you want to use them as condition to find traces, you should create indexes for them.
| Field Name | Type | Token |
|---|---|---|
| tags.http.method | text | N/A |
| tags.http.status_code | text | N/A |
Create logstore to store agg data and create the below index for it.（Optional, if you need to use span agg logstore to speed up the query of service and operation, you need to create it）
| Field Name | Type | Token |
|---|---|---|
| operationName | text | N/A |
| serviceName | text | N/A |
Jaeger client libraries expect jaeger-agent process to run locally on each host. The agent exposes the following ports:
| Port | Protocol | Function |
|---|---|---|
| 5775 | UDP | accept zipkin.thrift over compact thrift protocol |
| 6831 | UDP | accept jaeger.thrift over compact thrift protocol |
| 6832 | UDP | accept jaeger.thrift over binary thrift protocol |
| 5778 | HTTP | serve configs, sampling strategies |
If you have already installed docker, you can run agent as follows:
docker run \
--rm \
-p5775:5775/udp \
-p6831:6831/udp \
-p6832:6832/udp \
-p5778:5778/tcp \
jaegertracing/jaeger-agent:1.6.0 --collector.host-port=<JAEGER_COLLECTOR_HOST>:14267
If you have already built the corresponding binary file, take macOS as an example, you can run agent as follows:
./cmd/agent/agent-darwin --collector.host-port=localhost:14267
The collectors are stateless and thus many instances of jaeger-collector can be run in parallel. You need to specify the storage type used to store span. If you specify Aliyun Log Service as your backend storage, you also need to provide the relevant parameters for the log service.
Parameter Description
| Parameter Name | Type | Description | Is Optional | Default |
|---|---|---|---|---|
| SPAN_STORAGE_TYPE | environment variable | specify the storage type used to store span | N | n/a |
| aliyun-log.project | program argument | specify the project used to store span | N | n/a |
| aliyun-log.endpoint | program argument | specify the endpoint for your project | N | n/a |
| aliyun-log.access-key-id | program argument | specify the account information for your log services | N | n/a |
| aliyun-log.access-key-secret | program argument | specify the account information for your log services | N | n/a |
| aliyun-log.span-logstore | program argument | specify the logstore used to store span | N | n/a |
| aliyun-log.max-query-duration | program argument | specify the maximum query range. For example, --aliyun-log.max-query-duration=120h | N | n/a |
| aliyun-log.init-resource-flag | program argument | specify whether to init istio related resources | Y | true |
At default settings the collector exposes the following ports:
| Port | Protocol | Function |
|---|---|---|
| 14267 | TChannel | used by jaeger-agent to send spans in jaeger.thrift format |
| 14268 | HTTP | can accept spans directly from clients in jaeger.thrift format |
| 9411 | HTTP | can accept Zipkin spans in JSON or Thrift (disabled by default) |
If you have already installed docker, you can run collector as follows:
docker run \
-it --rm \
-p14267:14267 -p14268:14268 -p9411:9411 \
-e SPAN_STORAGE_TYPE=aliyun-log \
registry.cn-hangzhou.aliyuncs.com/jaegertracing/jaeger-collector:0.2.4 \
/go/bin/collector-linux \
--aliyun-log.project=<PROJECT> \
--aliyun-log.endpoint=<ENDPOINT> \
--aliyun-log.access-key-id=<ACCESS_KEY_ID> \
--aliyun-log.access-key-secret=<ACCESS_KEY_SECRET> \
--aliyun-log.span-logstore=<SPAN_LOGSTORE> \
--aliyun-log.init-resource-flag=false
If you have already built the corresponding binary file, take macOS as an example, you can run collector as follows:
export SPAN_STORAGE_TYPE=aliyun-log && \
./cmd/collector/collector-darwin \
--aliyun-log.project=<PROJECT> \
--aliyun-log.endpoint=<ENDPOINT> \
--aliyun-log.access-key-id=<ACCESS_KEY_ID> \
--aliyun-log.access-key-secret=<ACCESS_KEY_SECRET> \
--aliyun-log.span-logstore=<SPAN_LOGSTORE> \
--aliyun-log.init-resource-flag=false
jaeger-query serves the API endpoints and a React/Javascript UI. The service is stateless and is typically run behind a load balancer, e.g. nginx. Similar to collector, if you specify Aliyun Log Service as your backend storage, you also need to provide the relevant parameters for the log service. In addition, you need to specify the location of the UI static file by the parameter query.static-files.
Parameters Description
| Parameter Name | Type | Description | Is Optional | Default |
|---|---|---|---|---|
| SPAN_STORAGE_TYPE | environment variable | specify the storage type used to store span | N | n/a |
| aliyun-log.project | program argument | specify the project used to store span | N | n/a |
| aliyun-log.endpoint | program argument | specify the endpoint for your project | N | n/a |
| aliyun-log.access-key-id | program argument | specify the account information for your log services | N | n/a |
| aliyun-log.access-key-secret | program argument | specify the account information for your log services | N | n/a |
| aliyun-log.span-logstore | program argument | specify the logstore used to store span | N | n/a |
| aliyun-log.span-agg-logstore | program argument | specify the logstore used to store agg data | Y | "" |
| aliyun-log.init-resource-flag | program argument | specify whether to init istio related resources | Y | true |
| query.static-files | program argument | Specify the location of the UI static files | N | n/a |
At default settings the query service exposes the following port(s):
| Port | Protocol | Function |
|---|---|---|
| 16686 | HTTP | /api/* endpoints and Jaeger UI at / |
If you have already installed docker, you can run query as follows:
docker run \
-it --rm \
-p16686:16686 \
-e SPAN_STORAGE_TYPE=aliyun-log \
registry.cn-hangzhou.aliyuncs.com/jaegertracing/jaeger-query:0.2.4 \
/go/bin/query-linux \
--aliyun-log.project=<PROJECT> \
--aliyun-log.endpoint=<ENDPOINT> \
--aliyun-log.access-key-id=<ACCESS_KEY_ID> \
--aliyun-log.access-key-secret=<ACCESS_KEY_SECRET> \
--aliyun-log.span-logstore=<SPAN_LOGSTORE> \
--aliyun-log.span-agg-logstore=<SPAN_AGG_LOGSTORE> \
--aliyun-log.init-resource-flag=false \
--query.static-files=/go/jaeger-ui/
If you have already built the corresponding binary file, take macOS as an example, you can run query as follows:
export SPAN_STORAGE_TYPE=aliyun-log && \
./cmd/query/query-darwin \
--aliyun-log.project=<PROJECT> \
--aliyun-log.endpoint=<ENDPOINT> \
--aliyun-log.access-key-id=<ACCESS_KEY_ID> \
--aliyun-log.access-key-secret=<ACCESS_KEY_SECRET> \
--aliyun-log.span-logstore=<SPAN_LOGSTORE> \
--aliyun-log.span-agg-logstore=<SPAN_AGG_LOGSTORE> \
--aliyun-log.init-resource-flag=false \
--query.static-files=./jaeger-ui-build/build/
To simplify the deployment, we have provided a docker-compose template aliyunlog-jaeger-docker-compose.yml.
You can start jaeger-agent, jaeger-collector, and jaeger-query through the following commands
docker-compose -f aliyunlog-jaeger-docker-compose.yml up
You can stop jaeger-agent, jaeger-collector, and jaeger-query through the following commands
docker-compose -f aliyunlog-jaeger-docker-compose.yml stop
Note: please remember to replace the following parameters with the real value before you run the above commands.
Find traces
Display detailed information for trace
This project provide a demo applicatio named hotrod. Please refer to this doc.
- Alicloud Log Service homepage
- Alicloud Log Service doc
- Alicloud Log Servic official forum
- Alicloud Log Servic official technical support: submit tickets
@WPH95 made a great contribution to this project.
Thanks for the excellent work by @WPH95