add UltraLM-13b-V2.0/UltraLM-13b-V2.0-best-of-16/UltraLM-13b-best-of-16 to AlpacaEval #139
Conversation
| ultralm-13b: | ||
| prompt_template: "ultralm-13b-best-of-16/prompt.txt" | ||
| fn_completions: "huggingface_local_completions" | ||
| completions_kwargs: |
There was a problem hiding this comment.
did you not have to change the completion function to do best-of-16?
There was a problem hiding this comment.
Hi Yann,
I kept this "fn_completions" as other models since there is no instruction guiding what options are allowed in this field. Can I add a comment in the line, such as fn_completions: "huggingface_local_completions" # best-of-16 sampling? or can I change 'huggingface_local_completions' to something like 'huggingface_best_of_16_completions'?
There was a problem hiding this comment.
this is meant to be used to be able to replicate your results. All the options can be found here: https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/decoders/__init__.py
in you case, it seems that you cannot replicate your results with our code. Are all the parameters completions_kwargs correct then? GIven that you can't replicate your results with our code, I'd suggest just removing fn_completions and completions_kwargs, and adding a comment in the yml that says that results can't be reproduced in alpaca_eval because they require best of n sampling
|
Hi @lifan-yuan, can you upload the model outputs also? |
Sorry for the missing files. I will create a new commit adding the output files after we find out the right way to represent the best-of-16 sampling. |
|
Hi @YannDubs, I've added a comment in the yml file and explained how the results can be reproduced. The model_outputs are updated as well. |
|
LGTM! Great job! |
No description provided.