Fixed EVAL_PROMPT_TEMPLATE to handle empty string or multiple match answers

[jamesbraza]

This PR moves the evaluation prompt template to handle

• Empty string, which relates to Empty session.answer for exhaustive search without an answer  #723
• Multiple matches, which fixes Failures to correct evaluate non-single letter answers #709

[sidnarayanan]

How do the benchmarks move with this change?

[mskarlin]

I think this is a solid iteration of the eval prompt template to address #709.

But I don't think this addresses Empty session.answer for exhaustive search without an answer #723

We still need the ability to inform the user of what happened at this step -- they just get a blank answer object currently. If the "user" is the eval script (as here) then this takes care of it.

[jamesbraza]

Oh so you'd like to see something like the paper-qa runner script swapping out empty answer for some sentinel value? Can you elaborate where in the stack you'd like to see a fix, beyond LitQAEvaluation?

[mskarlin]

Yea, exactly -- after the runner finishes, we need some sentinel to be added. The blank isn't intuitive enough for users to understand (especially via command line or UI). Can def be another PR.

[mskarlin]

In cases like this eval though -- the sentinel will create a need for another prompt, so we could make the sentinel a module level variable and check for it in the eval method.

[jamesbraza]

I see what you're saying, am I correct that you took care of this in #726?

Regardless, this PR will no longer close #723

[jamesbraza]

How do the benchmarks move with this change?

I am going to be running this shortly. With the empty answers, previously sometimes they got marked as correct, sometimes incorrect, and sometimes unsure. I am not sure which is more frequent, but that will likely dictate the impact on benchmark performance.

Fwiw our mini test suite's benchmark has grown here.

[jamesbraza]

@sidnarayanan I will circle back with the impact of evaluation, going to merge this for now