EAGER: Income Learning: A New Model for Behavior-Analysis-Inspired Learning from Human Feedback

随着虚拟代理和物理机器人变得越来越普遍，它们可以有效地执行越来越多的复杂任务来帮助人类。这些任务通常被形式化为顺序决策任务，其中机器人和智能体感知状态，采取行动，并接收奖励反馈信号。在实践中，如果这些机器要完成原始开发预先指定的任务之外的任务，那么直接从人类用户那里学习是至关重要的。机器强化学习（RL）是一种经常用于解决顺序决策任务的范式，最初是在应用行为分析（阿坝）社区的动物学习研究的启发下开发的。现有的强化学习方法有效地操作了有限的阿坝原则;然而，阿坝研究中还有一些其他的原则和属性没有很好地封装在现有的强化学习形式主义中，这些原则和属性可能是设计更有效的强化学习技术的新灵感来源，这些技术能够向人类教师学习。该项目将（1）采用阿坝和RL的联合收割机原理，以产生可以更有效地从人类学习的算法，（2）在虚拟代理和机器人平台上评估这些算法，以及（3）调查非专家人类是否以及如何构建难度不断增加的任务序列，类似于专家动物训练师如何塑造任务。从这些用户研究中获得的见解将被用来进一步提高我们的算法向人类训练师学习的能力。一旦成功，这个项目将取得关键进展，使非技术用户能够教虚拟和物理代理在自然环境中执行复杂的任务，熟悉许多从以前的经验，在训练家庭宠物。这个项目是一个更大的努力之间的一部分，华盛顿州立大学（WSU），北卡罗来纳州州立大学，布朗大学。WSU的工作将集中在实现拟议的机器学习算法家族，称为收入学习（I-Learning）。由于这些算法是由三所大学共同开发的，WSU将设计用户研究来评估I-Learning背后的原则何时以及如何使其在从人类反馈中学习方面优于其他现有算法。WSU将主要专注于1）虚拟代理，允许通过众包进行测试学习，以及2）物理机器人的测试和研究，如果实施改变用户的感知和行动，或算法的学习效率。此外，华盛顿州立大学还将研究人性化的课程设计。专业的训练师可以塑造动物的行为，随着时间的推移增加任务的复杂性，这样动物就可以比直接训练最后的困难任务更快地学习一系列任务。WSU将在众包平台上进行用户研究，以更好地了解非专家人类如何在顺序决策任务中设计机器学习算法的课程，并研究这些设计决策如何为算法设计提供信息。

As virtual agents and physical robots become more common, there is an increasing number of complex tasks they can usefully perform to assist humans. These tasks are typically formalized as sequential decision tasks, where robots and agents perceive states, take actions, and receive a reward feedback signal. In practice, there is a critical need to learn directly from human users if such machines are to accomplish tasks outside of those pre-specified by the original developments. Machine reinforcement learning (RL), a paradigm often used for solving sequential decision making tasks, was originally developed with inspiration from animal learning research from the applied behavior analysis (ABA) community. Existing RL approaches operationalize a limited set of ABA principles effectively; however, there are additional principles and properties from ABA research that are not well encapsulated in the existing RL formalisms, and that are likely sources of new inspiration for designing more effective RL techniques capable of learning from human teachers. This project will (1) take combine principles from ABA and RL to produce algorithms that can learn more effectively from humans, (2) evaluate these algorithms in both virtual agents and on robot platforms, and (3) investigate whether and how non-expert humans can construct sequences of tasks of increasing difficulty, similar to how expert animal trainers shape tasks. Insights from these user studies will be leveraged to further improve our algorithms' abilities to learn from human trainers. Once successful, this project will make critical progress towards allowing non-technical users to be able to teach virtual and physical agents to perform complex tasks in a natural setting, familiar to many from previous experience in training household pets.This project is a part of a larger effort between Washington State University (WSU), North Carolina State University, and Brown University. The WSU effort will focus on implementing the proposed family of machine learning algorithms, called Income Learning (I-Learning). As these algorithms are co-developed by the three universities, WSU will design user studies to evaluate when and how the principles behind I-Learning allow it to outperform other existing algorithms at learning from human feedback. WSU will primarily focus on 1) virtual agents, allowing test learning via crowdsourcing, as well as testing on 2) physical robots and study if embodiment changes user's perceptions and actions, or the algorithms' learning efficacy. Additionally, WSU will investigate 3) human curricula design. Expert trainers can shape the behavior of animals, increasing task complexity over time, so that the animals can learn a sequence of tasks much faster than if they trained directly on the final, difficult task. WSU will run user studies on crowdsourcing platforms to better understand how non-expert humans design curricula for machine learning algorithms in sequential decision tasks, and investigate how these design decisions can inform algorithm design.