Feature Expansive Reward Learning: Rethinking Human Input
Feature Expansive Reward Learning: Rethinking Human Input
复制标题
扩展奖励学习功能：重新思考人类输入
DOI：
10.1145/3434073.3444667
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
A. Dragan
中科院分区：
文献类型：
--
作者：
Andreea Bobu;Marius Wiggert;C. Tomlin;A. Dragan
当人们对机器人执行任务的方式不满意时，他们可以进行干预以纠正它。奖励学习方法使机器人能够根据此类人类输入在线调整其奖励功能，但它们依赖于手工制作的功能。当这些特征无法解释校正时，深度逆强化学习（IRL）的最新研究表明，机器人可以要求任务演示并恢复在原始状态空间上定义的奖励。我们的见解是，机器人不应该从演示中隐式地了解缺失的功能，而应该要求明确地教导它缺失什么的数据。我们引入了一种新型的人类输入，其中人引导机器人从所教特征高度表达的状态到不高度表达的状态。我们提出了一种从原始状态空间学习特征并将其集成到奖励函数中的算法。通过将人类输入集中在缺失的特征上，我们的方法降低了样本复杂性，并提高了学习奖励在上述深度 IRL 基线上的泛化能力。我们在物理 7DOF 机器人操纵器的实验以及在模拟环境中进行的用户研究中展示了这一点。
When a person is not satisfied with how a robot performs a task, they can intervene to correct it. Reward learning methods enable the robot to adapt its reward function online based on such human input, but they rely on handcrafted features. When the correction cannot be explained by these features, recent work in deep Inverse Reinforcement Learning (IRL) suggests that the robot could ask for task demonstrations and recover a reward defined over the raw state space. Our insight is that rather than implicitly learning about the missing feature(s) from demonstrations, the robot should instead ask for data that explicitly teaches it about what it is missing. We introduce a new type of human input in which the person guides the robot from states where the feature being taught is highly expressed to states where it is not. We propose an algorithm for learning the feature from the raw state space and integrating it into the reward function. By focusing the human input on the missing feature, our method decreases sample complexity and improves generalization of the learned reward over the above deep IRL baseline. We show this in experiments with a physical 7DOF robot manipulator, as well as in a user study conducted in a simulated environment.
关闭开关游戏
DOI：
10.24963/ijcai.2017/32
发表时间：
2017
期刊：
International Joint Conferences on Artificial Intelligence Organization
影响因子：
--
作者：
Hadfield-Menell, Dylan;Dragan, Anca;Abbeel, Pieter;Russell, Stuart
通讯作者：
Russell, Stuart
影响因子：
7.8
作者：
Andreea Bobu;Andrea V. Bajcsy;J. Fisac;Sampada Deglurkar;A. Dragan
通讯作者：
Andreea Bobu;Andrea V. Bajcsy;J. Fisac;Sampada Deglurkar;A. Dragan