A proposal for integrating StreamAD, adding more anomaly detectors.

[Fengrui-Liu]

Hi community,

river is a wonderful project and I like it very much. However, I notice that its anomaly detectors only include 4 methods, and can be improved by adding more detectors.

If it is possible, I'd like to join online-ml organization, and integrate my repo StreamAD to river with PRs. StreamAD is a collection of open-source anomaly detectors, and it has similar APIs to river.

Be happy with any suggestions and comments 😄 .

[MaxHalford]

Hello there!

river is a wonderful project and I like it very much.

That's kind of you to say :)

However, I notice that its anomaly detectors only include 4 methods, and can be improved by adding more detectors.

I agree that there are likely essential methods which are missing and would fit nicely into River. However, we firmly believe that quality > quantity. Therefore the goal isn't to implement every method that exists, but rather to focus on the ones that are known to be performant. The less methods there are, the clearer things are for users, and the easier the situation for us poor maintainers.

If it is possible, I'd like to join online-ml organization, and integrate my repo StreamAD to river with PRs.

I think I speak for everyone by saying that we'd be more than happy to have you on board! We only for a soft commitment that you'll be ready to maintain the code that you contribute, and to answer questions that users might ask :).

StreamAD is a collection of open-source anomaly detectors, and it has similar APIs to river.

Indeed StreamAD looks rather similar to River. There are just a few API issues we need to find a solution to. In River we don't have a clear distinction between univariate and multivariate anomaly detection methods. You could say that univariate methods take as input a single value, as opposed to a dictionary as do multivariate methods. The issue is that then you can't compose these methods with other building blocks, such as StandardScaler. I'm thinking univariate methods should also process dictionaries. Either we enforce the dictionaries to have a length of 1, or to hold a specific key, or we aggregate the anomaly scores in some way: take the highest score, the average, or whatever. Anyway, stuff to think about. Suggestions are welcome :)

[Fengrui-Liu]

Thank you for your prompt reply 😄

Agree, quality should always be the primary concern. If so, I need to evaluate different algorithms extensively. After that, the selected detector will be contributed to river. Sounds like a lot of work to do 🤔 . I will try my best.

[MaxHalford]

So I'm not sure how you usually measure the performance of an anomaly detection model. I usually use an imbalanced classification dataset. I started adding so-called "tracks" to the evaluate module. It would be useful/nice to have an anomaly detection track to easily compare anomaly detection models.

[Fengrui-Liu]

I read the code of "track", time/memory/flops are great.

I notice that a series of classification performance metrics are used (AUC-ROC/Accuracy), according to this，time series aware performance metrics are seem to be more suitable for anomaly detection task. I'm going to have a try 👻

Another question is, is it necessary to make the performance evaluation in a real-time manner? With all my experience, the labeling process is manual (or semi-automatic), causing that the labeled datasets are static. Thus, evaluation is always performed offline, and the best model is selected to detect anomalies online. What's your opinion？

[MaxHalford]

I notice that a series of classification performance metrics are used (AUC-ROC/Accuracy), according to this，time series aware performance metrics are seem to be more suitable for anomaly detection task. I'm going to have a try 👻

I'm not savvy enough to suggest anything better! It made sense to me to simply use classification metrics. In particular, my experience working at companies is that ROCAUC is a good metric because it indicates how to anomaly detector is at ranking anomalies. And that has business value, because you want humans to check anomalies in order of priority.

Another question is, is it necessary to make the performance evaluation in a real-time manner? With all my experience, the labeling process is manual (or semi-automatic), causing that the labeled datasets are static. Thus, evaluation is always performed offline, and the best model is selected to detect anomalies online. What's your opinion？

Yes evaluation is mainly to be offline on a static dataset. With River, you should simply be able to use the evaluate.progressive_val_score function. Let me know if it's confusing!

[stefannesic]

Hello, I, too, really enjoyed using River while recently working in anomaly detection research for data streams. I would like to help by adding more anomaly detection algorithms. It would be nice to have a functioning version of Loda, RRCF and/or xStream in River (if author rights allow for it). Which algorithm would you prioritize ? I am here to help!

[MaxHalford]

Hey @stefannesic 👋

Thanks for offering your help. We'd be glad to collaborate with you!

So some students started working on xStream here. I'm not sure where they're at, but I suspect they won't be doing much progress now that there semester is over.

I'm not familiar with RRCF, but it seems very interesting. It shouldn't be too difficult to implement by using the existing bricks in River. For instance, one could reuse the logic in hst.py. I'd be glad to help out here in terms of code structure, if needed.

I think Loda would be great to have. I took some notes here, although I'm not sure it's exactly about Loda. I think Loda has a lot of cousins.

Sorry if this isn't very helpful in terms of choosing! In short, I think all 3 methods are useful. I would start with the method that seems easiest to implement. That way you familiarize yourself with the code base.

[stefannesic]

I think RRCF would be the simplest to implement for a beginner contributor to River, but I wouldn't mind taking over the xStream work. Let me look at the code once again and choose one of these two to implement.