Applying basic data science on Jaeger dataset

[Ashmita152]

Hello everyone,

I am an outreachy intern working under the guidance of @jpkrohling. As part of my internship, we are working on Jupyter notebooks for finding possible interesting insights in Jaeger's tracing dataset.

We wrote two notebooks to explore possibilities:

• Anomaly Detection for HTTP attacks.
• SLO Validation.

Repository: https://github.com/Ashmita152/jaeger-notebooks

I would appreciate your feedback and thoughts on possible improvements.

Regards.

[jpkrohling]

And we'd like to move this, along with the dataset repository to the jaegetracing organization, hoping that this will encourage people to extend this work.

[jkowall]

Thanks for making these, cool idea! I saw the work @pavolloffay did as well with the notebooks in the past.

How would someone use this? I can see in the notebook that you are pointing to a "dataset", but I assume you'd need to run this in some kind of batch way to analyze the data. This is very reactive, where you'd find out about issues a day or more later.

This should be implemented inside a stream processor. I know we have code here at logz.io which leverages Kafka streams to build the DAG on the fly, we will be open sourcing that soon. There is also the Spark component in Jaeger where this could be done in a stream. I actually see a better use case for analyzing this data and looking for patterns on the fly to notify the user or surface issues in the UI as being more immediately useful operationally.

[jpkrohling]

A blog post is coming, and you are right, streaming is definitely what we want for a production setup. For the context of the internship, given the limited amount of time, the proposal was to get a repository to host notebooks that could serve as entry point for people looking for this type of analysis.

[jkowall]

Makes sense, I think it's a cool concept for sure. I just think about the application of it to Jaeger and there is a big lift to get from A to B. I have a colleague who is working on various data science tasks and features and we could look at doing some more trace data analysis. The challenge is how would this fit into the Jaeger architecture and if people would be able to run it effectively. That's my concern.

[jpkrohling]

This notebooks repository wouldn't fit an existing architecture, as it does require data to be exported first. I think those two cases have different audiences: the notebooks are for data scientists who are looking to extract business value from the tracing data. The streaming would be for our typical "what's going on right now?" cases.

[yurishkuro]

Why do we need a new repo, can we not add them to what Pavol already has?

Regarding the application of notebook, they are certainly a power user tool. And they are more useful for proactive perf improvements that often require more thorough analysis than reactive work during an outage that really needs the investigative workflows built into the UI. But also notebooks are great for proofs of concept that can later be transitioned into streaming solutions.

[Ashmita152]

Hi Yuri,

I agree with you that this was just a proof-of-concept work and dedicating separate repository mayn't be worth. We can merge the work in jaeger-analytics-java repo but there are few things I am not sure about:

• Repository name has java in it. We can probably rename the repo to jaeger-analytics to overcome it.
• The repo looks like self sufficient with components like spark, grafana config files. I am not quite sure how this work will fit in there.

Thank you.

[jpkrohling]

ping @Ashmita152