v0.2.0 - Cache
Caching
Added two new Evaluators for handling caching, MemoryCache
and FileCache
.
Using the API:
evaluator = MemoryCache(SemanticEvaluator())
Using the CLI:
$ bench run --cache memory # or `file` or `none`
For commandline caching is on by default.
Match object
Changed the signature of evaluate_prediction
from bool
to return Optional[Evaluator.Match]
Match carries information about which of the expected
values matched with the output
from the tested model.
This will likely be extended in the next release with even more information.