Leveraging Human and Agent Guidance for Improved Reinforcement Learning

点击翻译按钮获取中文摘要

Reinforcement learning (RL) is a type of machine learning that lets virtual or physical agents learn through experience, often finding novel solutions to difficult problems and exceeding human performance. RL has had many exciting successes from video game playing to data center optimization. Unfortunately, there are still relatively few real-world, deployed, RL success stories. One reason is that learning a policy can be very slow and there is an emphasis on agents learning from scratch. In contrast, this research will better allow RL agents to learn from others. This research will enable more deployments of RL in real-world scenarios by using existing knowledge from humans and agents to jumpstart initial behavior and reach high performing policies more quickly. An RL agent student can receive help from a human or agent teacher with multiple kinds of guidance, such as demonstration, action advice, or direct reward feedback. When successful, this research will enable RL to be successfully deployed in more real-world scenarios by focusing costly exploration and jumpstarting initial behavior to quickly reach high quality policies. The goal of this student/teacher framework is to improve the student's learning (relative to learning without guidance) without harming the agent's final performance. An additional goal can be to have the student outperform the teacher. The research is divided into three specific aims. Aim 1 focuses on how agents can best use human guidance, when different types of guidance are more or less useful, and how humans want to provide guidance. Aim 2 considers when a student should ask for guidance, or when a teacher should proactively provide guidance. Aim 3 considers the more general case when a student can learn from multiple teachers and when multiple students can learn from a single teacher. A key criticism of RL is that it can be slow to learn and that initial performance can be poor. By leveraging other agents, programs, human experts, and human non-experts as teachers, this research will help create opportunities across industries where RL successfully learns in physical and virtual settings to impact people. Not only will this research program help create Canadian jobs by using RL to improve processes in existing companies, it may help RL create new opportunities for businesses and startups that do not currently exist. Graduate students involved in this research will develop critical research, machine learning, and human-AI interaction skills. Other research groups will benefit from developed software, as it will enable standardization and make human subject studies in RL more accessible.