SRPC supports reporting traces to OpenTelemetry

[holmes1412]

1. Introduction

SRPC supports generating and reporting tracing and spans, which can be reported in multiple ways, including exporting data locally or to OpenTelemetry.

Since SRPC follows the data specification of OpenTelemetry and the specification of w3c trace context, now we can use RPCSpanOpenTelemetry as the reporting plugin.

The report conforms to the Workflow style, which is pure asynchronous task and therefore has no performance impact on the RPC requests and services.

2. Usage

After the plugin RPCSpanOpenTelemetry is constructed, we can use add_filter() to add it into server or client.

For tutorial/tutorial-02-srpc_pb_client.cc, add 2 lines like the following :

int main()                                                                   
{                                                                        
    Example::SRPCClient client("127.0.0.1", 1412); 

    RPCSpanOpenTelemetry span_otel("http://127.0.0.1:55358"); // jaeger http collector ip:port   
    client.add_filter(&span_otel);
    ...

For tutorial/tutorial-01-srpc_pb_server.cc, add the similar 2 lines. We also add the local plugin to print the reported data on the screen :

int main()
{
    SRPCServer server;  

    RPCSpanOpenTelemetry span_otel("http://127.0.0.1:55358");                            
    server.add_filter(&span_otel);                                                 

    RPCSpanDefault span_log; // this plugin will print the tracing info on the screen                                                  
    server.add_filter(&span_log);                                              
    ...

make the tutorial and run both server and client, we can see some tracing information on the screen.

We can find the span_id: 04d070f537f17d00 in client become parent_span_id: 04d070f537f17d00 in server:

3. Traces on Jaeger

Open the show page of Jaeger, we can find our service name Example and method name Echo. Here are two span nodes, which were reported by server and client respectively.

As what we saw on the screen, the client reported span_id: 04d070f537f17d00 and server reported span_id: 00202cf737f17d00, these span and the correlated tracing information can be found on Jaeger, too.

4. About Parameters

How long to collect a trace, and the number of reported retries and other parameters can be specified through the constructor parameters of RPCSpanOpenTelemetry. Code reference: src/module/rpc_span_policies.h

The default value is to collect up to 1000 trace information per second, and features such as transferring tracing information through the srpc framework transparently have also been implemented, which also conform to the specifications.

5. Attributes

We can also use add_attributes() to add some other informations as OTEL_RESOURCE_ATTRIBUTES.

Please notice that our service name "Example" is set also thought this attributes, the key of which is service.name. If service.name is also provided in OTEL_RESOURCE_ATTRIBUTES by users, then srpc service name takes precedence. Refers to : OpenTelemetry#resource

6. Log and Baggage

SRPC provides log() and baggage() to carry some user data through span.

API :

void log(const RPCLogVector& fields);
void baggage(const std::string& key, const std::string& value);

As a server, we can use RPCContext to add log annotation:

class ExampleServiceImpl : public Example::Service                                 
{
public: 
    void Echo(EchoRequest *req, EchoResponse *resp, RPCContext *ctx) override
    {
        resp->set_message("Hi back");
        ctx->log({{"event", "info"}, {"message", "rpc server echo() end."}});
    }
};

As a client, we can use RPCClientTask to add log on span:

srpc::SRPCClientTask *task = client.create_Echo_task(...);
task->log({{"event", "info"}, {"message", "log by rpc client echo()."}});