Questions about AlpacaEval 2.0 evaluation

[lucasz05]

Hello, we encountered some issues while reproducing the test results in the paper. On the AlpacaEval 2.0, we noticed that your GitHub page stated that you followed the default settings and chose 'alpaca_eval_gpt4_turbo-fn' for evaluation. However, we found that the default annotators_config of AlpacaEval 2.0 was 'weighted_alpaca_eval_gpt4_turbo'. Therefore, we conducted experiments using the above two as annotators respectively. The lc_winrate result for 'weighted_alpaca_eval_gpt4_turbo' was 15.88. And the lc_winrate result for 'alpaca_eval_gpt4_turbo-fn' was 22.42! Upon further examination, we found that the preference ratings in 'alpaca_eval_gpt4_turbo-fn' are 1.0, 2.0, and 1.5. According to the introduction on the alpaca_eval author's GitHub page, this is the scoring method for AlpacaEval 1.0. Therefore, we doubt whether there is an issue here. We would like to know what reference and annotator you use in your experiments.

[lucasz05]

And we would like to know where you got the results about the Source LLMs in the paper, whether they were your experimental results or obtained from other places?