Learning from Active Human Involvement through Proxy Value Propagation
Learning from Active Human Involvement through Proxy Value Propagation
复制标题
DOI：
--
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Zhenghao Peng;Wenjie Mo;Chenda Duan;Quanyi Li;Bolei Zhou
中科院分区：
文献类型：
--
作者：
Zhenghao Peng;Wenjie Mo;Chenda Duan;Quanyi Li;Bolei Zhou
从积极的人类参与中学习，使人类主体能够在训练期间积极干预并向AI代理演示。来自人类的交互和纠正反馈为学习过程带来了安全和AI对齐。在这项工作中，我们提出了一个新的无奖励的积极的人类参与方法称为代理值传播的政策优化。我们的关键见解是，可以设计一个代理值函数来表达人类的意图，其中人类演示中的状态-动作对被标记为高值，而那些被干预的代理的动作则被标记为低值。通过TD-学习框架，被证明的状态-动作对的标记值被进一步传播到从代理的探索产生的其他未标记的数据。因此，代理值函数诱导了忠实地模仿人类行为的策略。人在回路实验表明，我们的方法的通用性和效率。通过对现有强化学习算法的最小修改，我们的方法可以学习解决各种人类控制设备的连续和离散控制任务，包括在侠盗猎车手V中驾驶的挑战性任务。
Learning from active human involvement enables the human subject to actively intervene and demonstrate to the AI agent during training. The interaction and corrective feedback from human brings safety and AI alignment to the learning process. In this work, we propose a new reward-free active human involvement method called Proxy Value Propagation for policy optimization. Our key insight is that a proxy value function can be designed to express human intents, wherein state-action pairs in the human demonstration are labeled with high values, while those agents’ actions that are intervened receive low values. Through the TD-learning framework, labeled values of demonstrated state-action pairs are further propagated to other unlabeled data generated from agents’ exploration. The proxy value function thus induces a policy that faithfully emulates human behaviors. Human-in-the-loop experiments show the generality and efficiency of our method. With minimal modification to existing reinforcement learning algorithms, our method can learn to solve continuous and discrete control tasks with various human control devices, including the challenging task of driving in Grand Theft Auto V.