n_shots field creates fewshot queries with n+1 examples

[Samarendra109]

In the create_fewshot_query inside make_nshot_dataset (In genbench.task.py file), we have the following line of code,

in_context_example_ids = rng.choice(len(fewshot_example_source), num_shots + 1, replace=False).tolist()
in_context_examples = [ fewshot_example_source[i] for i in in_context_example_ids if fewshot_example_source[i] != query ]

Now this code will give correct fewshot query when the fewshot datasource is the same as the test suite and the current question is contained in the randomly selected shots. But when that is not the case, this will create a fewshot prompt with n+1 examples.

---

Additionally, the following line in make_nshot_dataset should also be fixed,

if len(fewshot_example_source) < num_shots + 1:
raise ValueError("Not enough examples for the number of shots.")

as this line is only valid if the fewshot_example_source is the same as the test dataset.

[Samarendra109]

I want to take up the issue.