feat: Evaluate missing splits

[isaac-chung]

Fixes #1260

• Based on fix: evaluate missing splits #1268 and addresses most of the comments
• Once run, the returned result will contain the missing splits as well as the already run splits
• Requires overwrite=True to work
• Changed tests to use the same MTEB object instead of a new one

Example Usage

from mteb import MTEB, get_tasks, get_model

tasks = get_tasks(tasks=["Banking77Classification"])
model = get_model("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")

# Runs only one split
evaluation = MTEB(tasks=tasks)
results = evaluation.run(
    model,
    eval_splits=["val"],
    output_folder=str(tmp_path / "testcase2"),
    verbosity=2,
)

# Will run the missing split and return results of both splits.
results2 = evaluation.run(
    model,
    eval_splits=["val", "test"],
    output_folder=str(tmp_path / "testcase2"),
    verbosity=2,
    overwrite_results=True,
)

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

[Samoed]

Overall looks great!

[Samoed]

What we should do if existing result and new result have different version?

[KennethEnevoldsen]

One solution is only to extend results if the versions match.

[isaac-chung]

only to extend results if the versions match.

This sounds like a natural line in the sand for now, at least for key versions where the results object are drastically different (e.g. pre-1.11.0). I can open an improvement issue to handle results from different versions?

[edit]: Hmm doesn't TaskResult.from_disk handle version difference already? There are methods like _convert_from_before_v1_11_0 and checks for pre_v_12_48.

[Samoed]

Then maybe to use verion from new_results? Because it will dump in format of running version

[isaac-chung]

Sure, will do.

Separately, I don't think this currently takes the difference in dataset version into consideration. I think this is already an existing gap, where we only check whether the same model + model revision has been run, but not check for dataset version. We should probably address it here before merging. wdyt?

[Samoed]

Most datasets downloaded by revision, so I don't think we need more checks

[isaac-chung]

Ok, then this should be good to merge once the tests pass.

[KennethEnevoldsen]

I think it is good to merge. It might be nice to create a "deprecation version", e.g. before 1.11.0, we can then slowly outdated old results as needed. However, this is probably something for a more general discussion, before any implementation.

[KennethEnevoldsen]

Looks great only a few minor things

[isaac-chung]

Thanks @Samoed and @KennethEnevoldsen for reviewing, and @thivyanth for the initial iteration! Merging now.