EAGER: Formal Models of Trainer Feedback for I-Learning Theoretical Guarantees

EAGER：I-Learning 理论保证的培训师反馈正式模型

• 批准号: 1643411
• 负责人: David Roberts
• 金额: $ 7万
• 依托单位: North Carolina State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-08-15 至 2017-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1643411&HistoricalAwards=false
• 关键词: EAGER; Formal; Models; Trainer; Feedback

As virtual agents and physical robots become more common, there is an increasing number of complex tasks they can usefully perform to assist humans. These tasks are typically formalized as sequential decision tasks, where robots and agents perceive states, take actions, and receive a reward feedback signal. In practice, there is a critical need to learn directly from human users---the majority of human users will not be able to directly program or fully specify a useful reward function. On the other hand, they can likely train an agent to perform tasks unanticipated by the original designer. Machine reinforcement learning (RL), a paradigm often used for solving sequential decision making tasks, was originally developed with inspiration from animal learning research from the applied behavior analysis (ABA) community. Existing RL approaches operationalize a limited set of ABA principles effectively; however, there are additional principles and properties from ABA research that are not well encapsulated in the existing RL formalisms, and that are likely sources of new inspiration for designing more effective RL techniques capable of learning from human teachers. The objective of this project is to leverage insights from animal training to reformulate the learning of sequential tasks from an agent learning alone in a fixed environment to an agent learning cooperatively with a competent, but not necessarily perfect, human teacher. Successful completion of this project will contribute a foundation of knowledge that will aide in the development new technologies to allow end users to customize the functions of their gadgets. This project is a part of a larger and collaborative effort between North Carolina State University (NCSU), Brown University, and Washington State University (WSU). The NCSU effort will include theoretical contributions along with empirical analyses and data collection. The emphasis of the NCSU portion of the project will be on the development of theoretical models of human feedback. When humans provide rewards to learning machines, describing the properties of the algorithms those machines use requires knowledge of how the humans provide feedback. For example, knowing when and how they make errors, the circumstances where they provide reinforcement or punishment, or use extinction, etc. Understanding the theoretical properties of I-Learning under different trainer paradigms will be the primary effort of NCSU project personnel. NCSU personnel will also work in concert with collaborators at Brown to use these models of feedback for describing the performance properties of I-Learning under different assumptions of trainer behavior. In addition, NCSU personnel will work with WSU collaborators to collect data from human trainers in virtual settings in order to validate and set the parameters of the theoretical models.

随着虚拟代理和物理机器人变得越来越普遍，它们可以有效地执行越来越多的复杂任务来帮助人类。这些任务通常被形式化为顺序决策任务，其中机器人和代理感知状态、采取行动并接收奖励反馈信号。在实践中，迫切需要直接从人类用户那里学习-大多数人类用户将无法直接编程或完全指定有用的奖励功能。另一方面，他们可能会训练一名代理执行原始设计者没有预料到的任务。机器强化学习(RL)是一种经常用于解决顺序决策任务的范式，最初是受应用行为分析(ABA)社区的动物学习研究的启发而发展起来的。现有的RL方法有效地操作了一组有限的ABA原则；然而，来自ABA研究的其他原则和性质没有很好地封装在现有的RL形式中，这些可能是设计能够从人类教师那里学习的更有效的RL技术的新灵感的来源。这个项目的目标是利用动物训练的洞察力来重新制定顺序任务的学习，从一个单独在固定环境中学习的代理学习到一个有能力但不一定完美的人类教师合作学习。该项目的成功完成将有助于开发新技术，使最终用户能够定制其小工具的功能，从而为开发新技术奠定基础。该项目是北卡罗来纳州立大学(NCSU)、布朗大学(Brown University)和华盛顿州立大学(WSU)之间更大规模的合作努力的一部分。NCSU的工作将包括理论贡献以及经验分析和数据收集。该项目NCSU部分的重点将是开发人类反馈的理论模型。当人类向学习机器提供奖励时，描述这些机器使用的算法的属性需要了解人类如何提供反馈。例如，了解他们何时以及如何犯错，他们在什么情况下提供强化或惩罚，或使用灭绝等。了解不同教员范式下i-Learning的理论属性将是NCSU项目人员的主要工作。NCSU人员还将与Brown的合作者合作，使用这些反馈模型来描述i-Learning在不同培训师行为假设下的性能属性。此外，NCSU人员将与WSU合作者合作，在虚拟环境中从人类教练员那里收集数据，以验证和设置理论模型的参数。

项目成果

A Need for Speed: Adapting Agent Action Speed to Improve Task Learning from Non-Expert Humans

对速度的需求：调整代理动作速度以提高非专家的任务学习能力

• 发表时间: 2016
• 期刊: Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems
• 作者: Peng, Bei;MacGlashan, James;Loftin, Robert;Littman, Michael L.;Roberts, David L.;Taylor, Matthew E.
• 通讯作者: Taylor, Matthew E.