RRHF: Rank Responses to Align Language Models with Human Feedback without tears
RRHF: Rank Responses to Align Language Models with Human Feedback without tears
复制标题
DOI：
10.48550/arxiv.2304.05302
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Zheng Yuan;Hongyi Yuan;Chuanqi Tan;Wei Wang;Songfang Huang;Feiran Huang
中科院分区：
文献类型：
--
作者：
Zheng Yuan;Hongyi Yuan;Chuanqi Tan;Wei Wang;Songfang Huang;Feiran Huang
基于人类反馈的强化学习（RLHF）促进了大型语言模型与人类偏好的对齐，显著提高了人类与模型之间的交互质量。InstructGPT通过几个阶段实现RLHF，包括监督微调（SFT）、奖励模型训练和近端策略优化（PPO）。然而，PPO对超参数很敏感，并且在其标准实现中需要多个模型，这使得它很难训练和扩展到更大的参数计数。相比之下，我们提出了一种新的学习范式，称为RRHF，它通过条件概率的对数对来自不同来源的采样响应进行评分，并通过排序损失来学习将这些概率与人类偏好保持一致。RRHF可以利用来自各种来源的采样响应，包括来自自身的模型响应、其他大型语言模型响应和人类专家响应，以学习对它们进行排序。RRHF在调优过程中只需要1到2个模型，无需复杂的超参数调优，就能有效地将语言模型与人类偏好进行鲁棒对齐。此外，RRHF可以被认为是SFT和奖励模型训练的扩展，同时在编码、模型计数和超参数方面比PPO更简单。我们在有用和无害数据集上评估了RRHF，通过奖励模型得分和人类标记证明了与PPO相当的对齐性能。大量的实验表明，RRHF的性能与采样质量高度相关，这表明RRHF是一种best-of-n学习器。代码可在https://github.com/GanjinZero/RRHF获得。
Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). However, PPO is sensitive to hyperparameters and requires multiple models in its standard implementation, making it hard to train and scale up to larger parameter counts. In contrast, we propose a novel learning paradigm called RRHF, which scores sampled responses from different sources via a logarithm of conditional probabilities and learns to align these probabilities with human preferences through ranking loss. RRHF can leverage sampled responses from various sources including the model responses from itself, other large language model responses, and human expert responses to learn to rank them. RRHF only needs 1 to 2 models during tuning and can efficiently align language models with human preferences robustly without complex hyperparameter tuning. Additionally, RRHF can be considered an extension of SFT and reward model training while being simpler than PPO in terms of coding, model counts, and hyperparameters. We evaluate RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment performance with PPO by reward model score and human labeling. Extensive experiments show that the performance of RRHF is highly related to sampling quality which suggests RRHF is a best-of-n learner. Codes available at https://github.com/GanjinZero/RRHF.