Best detection data set(s)?

[scottveirs]

An applied conservation problem is detecting the presence of SRKWs as early as possible in live audio data streams. What algorithm or combinations of algorithms can optimally detect SRKW signals, whether they be calls, clicks, whistles, &/or other sounds (e.g. percussives)? Can a new strategy match or exceed the current performance of the OrcaHello real-time inference system?

Let’s discuss what open data set(s) could be offered through DCLDE to develop and test new approaches.

[scottveirs]

One data set is the archived Orcasound bioacoustics bouts for which there was at least one machine or human detection in real time. The beginning of each bout could be reviewed by domain experts who manually determine the first detectable signal above some confidence threshold. Then lag between that time and the first confident detection of any given technique could be assessed, and compared with the real-time performance of the production model and listeners.

[scottveirs]

@JPalmerK This is the idea I mentioned today for a real time audio data challenge for DCLDE. The ultimate goal would be a deployed model that is consistently (or most often) the first to detect SRKWs with a confidence above some threshold and a false positive rate of less than e.g. 10 per day. A bonus for Orcasound live hydrophones would be providing such detections to the community via the emerging Orcasound backend API -- both for moderation by trained experts, but also for sharing with other members of the listening community and comparison with other models (in friendly competition).

As you rightly point out, a stepping stone to this ultimate deployment of new open models "as a detection service" and API-based programatic exchanges of model outputs could be holding out a few test sets until the last few months leading up to the DCLDE workshop. I can provide a few such sets in which the first SRKWs signals of a bout, often with very low SNR or "tricky" calls, are annotated by human experts with high-confidence.

[bnestor]

Hi Scott and others,
I created a multiple instance learning setup for orcas from orcasound data and ONC. I have some models that are getting ~0.95 AUROC and ~0.9 AUPR. I have ~1100 hours of whale chatter thus far, mostly covering the SRKWs (about 10% are humpbacks and an unknown quantity of transients). I am working on hosting to share the curated dataset once I finish up searching the ONC archive. Data are under CC license for ONC and CC-NC for orcasounds data. I would also check out ORCA-SPOT and ANIMAL-SPOT, which were trained on northern residents. There are also some interesting techniques from the DCASE bioacoustic detection challenge that would be useful to apply to orca data.