EAGER: Income Learning: A New Model for Behavior-Analysis-Inspired Learning from Human Feedback

EAGER：收入学习：基于人类反馈的行为分析启发学习的新模型

• 批准号: 1643614
• 负责人: Matthew Taylor
• 金额: $ 7万
• 依托单位: Washington State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-08-15 至 2017-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1643614&HistoricalAwards=false
• 关键词: EAGER; Income; Learning; New; Model

As virtual agents and physical robots become more common, there is an increasing number of complex tasks they can usefully perform to assist humans. These tasks are typically formalized as sequential decision tasks, where robots and agents perceive states, take actions, and receive a reward feedback signal. In practice, there is a critical need to learn directly from human users if such machines are to accomplish tasks outside of those pre-specified by the original developments. Machine reinforcement learning (RL), a paradigm often used for solving sequential decision making tasks, was originally developed with inspiration from animal learning research from the applied behavior analysis (ABA) community. Existing RL approaches operationalize a limited set of ABA principles effectively; however, there are additional principles and properties from ABA research that are not well encapsulated in the existing RL formalisms, and that are likely sources of new inspiration for designing more effective RL techniques capable of learning from human teachers. This project will (1) take combine principles from ABA and RL to produce algorithms that can learn more effectively from humans, (2) evaluate these algorithms in both virtual agents and on robot platforms, and (3) investigate whether and how non-expert humans can construct sequences of tasks of increasing difficulty, similar to how expert animal trainers shape tasks. Insights from these user studies will be leveraged to further improve our algorithms' abilities to learn from human trainers. Once successful, this project will make critical progress towards allowing non-technical users to be able to teach virtual and physical agents to perform complex tasks in a natural setting, familiar to many from previous experience in training household pets.This project is a part of a larger effort between Washington State University (WSU), North Carolina State University, and Brown University. The WSU effort will focus on implementing the proposed family of machine learning algorithms, called Income Learning (I-Learning). As these algorithms are co-developed by the three universities, WSU will design user studies to evaluate when and how the principles behind I-Learning allow it to outperform other existing algorithms at learning from human feedback. WSU will primarily focus on 1) virtual agents, allowing test learning via crowdsourcing, as well as testing on 2) physical robots and study if embodiment changes user's perceptions and actions, or the algorithms' learning efficacy. Additionally, WSU will investigate 3) human curricula design. Expert trainers can shape the behavior of animals, increasing task complexity over time, so that the animals can learn a sequence of tasks much faster than if they trained directly on the final, difficult task. WSU will run user studies on crowdsourcing platforms to better understand how non-expert humans design curricula for machine learning algorithms in sequential decision tasks, and investigate how these design decisions can inform algorithm design.

随着虚拟代理和物理机器人变得越来越普遍，它们可以有效地执行越来越多的复杂任务来帮助人类。这些任务通常形式化为顺序决策任务，其中机器人和代理感知状态，采取行动并接收奖励反馈信号。在实践中，如果这些机器要完成原始开发预先指定的任务之外的任务，则迫切需要直接向人类用户学习。机器强化学习（RL）是一种通常用于解决顺序决策任务的范式，最初是在应用行为分析（ABA）社区的动物学习研究的启发下开发的。现有的强化学习方法有效地实现了一套有限的ABA原则；然而，ABA研究中还有一些额外的原则和特性并没有很好地封装在现有的强化学习形式中，这可能是设计能够向人类教师学习的更有效的强化学习技术的新灵感来源。该项目将(1)将ABA和RL的原理结合起来，产生可以更有效地向人类学习的算法，(2)在虚拟代理和机器人平台上评估这些算法，以及(3)研究非专业人类是否以及如何构建难度越来越大的任务序列，类似于专业动物驯兽师如何塑造任务。从这些用户研究中获得的见解将被用来进一步提高我们的算法向人类训练师学习的能力。一旦成功，该项目将在允许非技术用户能够教虚拟和物理代理在自然环境中执行复杂任务方面取得关键进展，许多人从以前训练家庭宠物的经验中熟悉这些任务。该项目是华盛顿州立大学（WSU）、北卡罗来纳州立大学和布朗大学之间更大合作的一部分。华盛顿州立大学的工作将集中在实施提议的机器学习算法家族，称为收入学习（I-Learning）。由于这些算法是由三所大学共同开发的，华盛顿州立大学将设计用户研究，以评估I-Learning背后的原理何时以及如何使其在从人类反馈中学习方面优于其他现有算法。WSU将主要关注1)虚拟代理，允许通过众包进行测试学习，以及2)物理机器人的测试，并研究具体化是否会改变用户的感知和行为，或者算法的学习效率。此外，华盛顿州立大学将研究人文课程设计。专业的训练师可以塑造动物的行为，随着时间的推移增加任务的复杂性，这样动物就能比直接接受最终的、困难的任务训练更快地学会一系列任务。WSU将在众包平台上进行用户研究，以更好地了解非专业人员如何为顺序决策任务中的机器学习算法设计课程，并研究这些设计决策如何为算法设计提供信息。