Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENH] Implement range-based precision and recall within aeon (replacing prts package) #2066

Open
3 tasks
SebastianSchmidl opened this issue Sep 17, 2024 · 3 comments
Assignees
Labels
anomaly detection Anomaly detection package dependencies This issue or PR is related to dependencies of aeon enhancement New feature, improvement request or other non-bug code enhancement implementing algorithms Implementing new algorithms/estimators

Comments

@SebastianSchmidl
Copy link
Member

SebastianSchmidl commented Sep 17, 2024

Describe the feature or idea you want to propose

The package prts is used in the performance evaluation module for anomaly detection as a soft-dependency. However, there have not been any changes to it for 3 years, the maintainers have not reacted to my PR, and the package was not implemented by the original authors of the method/paper.

Describe your proposed solution

We should implement the metrics within aeon directly:

  • ts_fscore
  • ts_precision
  • ts_recall

Target module: aeon.benchmarking.metrics.anomaly_detection
Original publication:

Tatbul, Nesime, Tae Jun Lee, Stan Zdonik, Mejbah Alam, and Justin Gottschlich. "Precision and Recall for Time Series." In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS), 1920–30. 2018.
http://papers.nips.cc/paper/7462-precision-and-recall-for-time-series.pdf.

Describe alternatives you've considered, if relevant

not relevant

Additional context

No response

@SebastianSchmidl SebastianSchmidl added enhancement New feature, improvement request or other non-bug code enhancement anomaly detection Anomaly detection package dependencies This issue or PR is related to dependencies of aeon labels Sep 17, 2024
@SebastianSchmidl SebastianSchmidl changed the title [ENH] [ENH] Implement range-based precision and recall within aeon (replacing prts package) Sep 17, 2024
@TonyBagnall TonyBagnall added the implementing algorithms Implementing new algorithms/estimators label Sep 18, 2024
@aryanpola
Copy link
Contributor

@aeon-actions-bot assign @aryanpola

@aryanpola
Copy link
Contributor

@CodeLionX So, the true anomaly ranges would use the timepoints to create a range to compare with the algorithm-detected anomaly range. Maybe later find an intersection between them.

@SebastianSchmidl
Copy link
Member Author

Hi @aryanpola,
yes and no. Please read the paper carefully, it is quite detailed in its descriptions on how to compute ts_precision and ts_recall respectively (math formulas and pseudocode).

In their formulation, they convert both the ground truth (point) values and the (point) predictions into ranges, and then iterate over the real/true anomaly ranges for recall, but over the predicted anomaly ranges for precision.

You can use the existing implementation in prts to generate test cases to validate that your implementation is doing the same.

@aryanpola aryanpola mentioned this issue Nov 26, 2024
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
anomaly detection Anomaly detection package dependencies This issue or PR is related to dependencies of aeon enhancement New feature, improvement request or other non-bug code enhancement implementing algorithms Implementing new algorithms/estimators
Projects
None yet
Development

No branches or pull requests

3 participants