Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
复制标题
DOI：
10.48550/arxiv.2306.05685
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Lianmin Zheng;Wei-Lin Chiang;Ying Sheng;Siyuan Zhuang;Zhanghao Wu;Yonghao Zhuang;Zi Lin;Zhuohan Li;Dacheng Li;E. Xing;Haotong Zhang;Joseph Gonzalez;Ion Stoica
中科院分区：
文献类型：
--
作者：
Lianmin Zheng;Wei-Lin Chiang;Ying Sheng;Siyuan Zhuang;Zhanghao Wu;Yonghao Zhuang;Zi Lin;Zhuohan Li;Dacheng Li;E. Xing;Haotong Zhang;Joseph Gonzalez;Ion Stoica
由于其广泛的能力以及现有基准在衡量人类偏好方面的不足，因此评估基于大语言模型（LLM）的聊天助手是具有挑战性的。为了解决这个问题，我们探索使用强大的LLM作为法官来评估这些模型在更开放的问题上。我们研究了法学硕士法官的使用和局限性，包括位置，冗长和自我增强偏见以及有限的推理能力，并提出解决方案来减轻其中的某些解决方案。然后，我们通过引入两个基准来验证LLM法官与人类偏好之间的共识：MT Bench，一个多转变的问题集；还有众包战斗平台Chatbot Arena。我们的结果表明，强大的LLM法官（例如GPT-4）可以很好地匹配受控和众包人类的偏好，达到超过80％的一致性，这是人类之间的同意水平。因此，llm-as-a-a-gudge是一种可扩展且可解释的方式，可以近似人类的偏好，否则它的获得非常昂贵。此外，我们通过评估多种骆驼和维库纳（Vicuna）的变体来展示我们的基准和传统基准相互补充。 https://github.com/lm-sys/fastChat/fastChat/main/main/main/main/fastchat/llm_judge公开获得了MT-Bench问题，3K专家票以及与人类偏好的30k对话。
Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.