Leveraging Human and Agent Guidance for Improved Reinforcement Learning

强化学习（RL）是一种机器学习，它让虚拟或物理代理通过经验学习，经常找到解决难题的新方法，并超越人类的表现。从视频游戏到数据中心优化，强化学习取得了许多令人兴奋的成功。不幸的是，现实世界中部署的强化学习成功案例仍然相对较少。一个原因是学习策略可能非常缓慢，并且强调代理从头开始学习。相比之下，这项研究将更好地允许强化学习代理向他人学习。这项研究将通过使用来自人类和代理的现有知识来启动初始行为并更快地达到高性能策略，从而使强化学习在现实场景中得到更多的部署。RL代理学生可以从人类或代理教师那里获得多种指导，例如演示、行动建议或直接奖励反馈。一旦成功，这项研究将使强化学习能够成功地部署在更多的现实场景中，专注于昂贵的探索和快速启动初始行为，以快速达到高质量的策略。这个学生/教师框架的目标是提高学生的学习（相对于没有指导的学习），而不损害代理的最终表现。另一个目标是让学生的表现超过老师。本研究分为三个具体目标。Aim 1的重点是智能体如何最好地利用人类的引导，当不同类型的引导或多或少有用，以及人类希望如何提供指导。目标2考虑学生何时应该寻求指导，或者教师何时应该主动提供指导。目标3考虑了更一般的情况，即一个学生可以向多个老师学习，以及多个学生可以向一个老师学习。对强化学习的一个关键批评是，它的学习速度很慢，最初的表现可能很差。通过利用其他代理、程序、人类专家和人类非专家作为教师，这项研究将有助于为强化学习在物理和虚拟环境中成功学习以影响人们的行业创造机会。这个研究项目不仅可以通过使用RL来改善现有公司的流程来帮助创造加拿大的就业机会，还可以帮助RL为目前不存在的企业和初创公司创造新的机会。参与这项研究的研究生将发展批判性研究、机器学习和人类与人工智能的互动技能。其他研究小组将从开发的软件中受益，因为它将实现标准化，并使强化学习中的人类受试者研究更容易获得。

Reinforcement learning (RL) is a type of machine learning that lets virtual or physical agents learn through experience, often finding novel solutions to difficult problems and exceeding human performance. RL has had many exciting successes from video game playing to data center optimization. Unfortunately, there are still relatively few real-world, deployed, RL success stories. One reason is that learning a policy can be very slow and there is an emphasis on agents learning from scratch. In contrast, this research will better allow RL agents to learn from others. This research will enable more deployments of RL in real-world scenarios by using existing knowledge from humans and agents to jumpstart initial behavior and reach high performing policies more quickly. An RL agent student can receive help from a human or agent teacher with multiple kinds of guidance, such as demonstration, action advice, or direct reward feedback. When successful, this research will enable RL to be successfully deployed in more real-world scenarios by focusing costly exploration and jumpstarting initial behavior to quickly reach high quality policies. The goal of this student/teacher framework is to improve the student's learning (relative to learning without guidance) without harming the agent's final performance. An additional goal can be to have the student outperform the teacher. The research is divided into three specific aims. Aim 1 focuses on how agents can best use human guidance, when different types of guidance are more or less useful, and how humans want to provide guidance. Aim 2 considers when a student should ask for guidance, or when a teacher should proactively provide guidance. Aim 3 considers the more general case when a student can learn from multiple teachers and when multiple students can learn from a single teacher. A key criticism of RL is that it can be slow to learn and that initial performance can be poor. By leveraging other agents, programs, human experts, and human non-experts as teachers, this research will help create opportunities across industries where RL successfully learns in physical and virtual settings to impact people. Not only will this research program help create Canadian jobs by using RL to improve processes in existing companies, it may help RL create new opportunities for businesses and startups that do not currently exist. Graduate students involved in this research will develop critical research, machine learning, and human-AI interaction skills. Other research groups will benefit from developed software, as it will enable standardization and make human subject studies in RL more accessible.