Suggestion: Report the not yet implemented correctness metrics results to be based on 0 measurements in reports

[wtoorop]

Currently doing the queries for the Correctness metrics are implemented, and aggregation of the processed measurements is implemented, but processing the Correctness measurements is not yet fully implemented. That this is still a work in progress, which is clearly mentioned in the README.md.

Because aggregation is already implemented for all metrics (including Correctness) in report_creator.py, the reported correctness metrics currently always shows up as 100% correct, even though they are not actually performed. Perhaps for clarity they should be left out in reports (or should only enabled with a flag) until implementation of the correctness measurement is done. What do you think?

[paulehoffman]

RSSAC047 says that reports will include correctness. Instead of not reporting the (bogus) 100%, it would be better if the reports say something like "This is being reported as required by RSSAC047, but the result is possibly wrong". Please consult your sponsors on that wording.

[wtoorop]

RSSAC047 says that reports will include correctness. Instead of not reporting the (bogus) 100%, it would be better if the reports say something like "This is being reported as required by RSSAC047, but the result is possibly wrong". Please consult your sponsors on that wording.

In this case I want to discuss the RSSAC047v2 initial implementation with the implementer. I don't want to change RSSAC047v2 directions, I'm merely noticing that an untruth is being reported. I believe reporting an untruth is also not following RSSAC047v2 intentions.

One way to still report on the Correctness metrics is to report "pass" but report 0 as the number of measurements on which that is based. That would match reality and also follow RSSAC047v2 to the letter. What do you think?

[paulehoffman]

Reporting the untruth is indeed not following the intention of RSSAC047, but I think not including the false report is further from the requirements.

I like the idea of reporting the number of measurements as 0!