Reporting results with NAB

In order for a researcher to report NAB results on an algorithm, it must meet the following additional criteria:

• Any sort of look ahead is not allowed. Detections by the algorithm must be made on the fly, one record at a time.

• There must be no batch, or per dataset, parameter tuning. The algorithm must be fully automated with a single set of parameters across all data files. Any further parameter tuning required by the algorithm must be done on the fly. (One exception to this rule is that it is ok for the algorithm to know the range of data values for each data file. In the NAB detector code, this is passed in automatically as each algorithm is initialized.)

• Each file is treated independently. The algorithm is initialized from scratch for each stream using a single constant set of parameters.

These rules are designed to simulate real-world streaming scenarios. It is very easy to "game" the system without following the above rules. Anyone reporting NAB results should convince the reader that the above criteria are met. A great way is to release all your code as open source, or by documenting your process sufficiently so that someone else can replicate it.