Immersive Visualization of Reinforcement Learning

Borealis AI是加拿大皇家银行（RBC）研究所，致力于强化学习（RL）、深度学习、对抗性机器学习和其他相关领域的基础和应用研究。除了应用最先进的方法外，他们还开发了专门用于解决这些现实挑战的新强化学习技术。该框架的一个基本改进是纳入了人类知识——利用用户的能力可以帮助系统在广泛的任务中更快更好地学习。这项研究的长期目标是从显性和隐性用户对系统的反馈中学习。为了在实现这一目标方面取得进展，必须克服两个挑战。首先，非技术培训师必须能够理解代理人的政策或行为方式。我们假设，培训师越了解代理知道什么和不知道什么，她就越能提供更好的帮助。其次，Borealis人工智能强化学习专家必须更好地理解训练师的输入是如何影响算法的。我们假设，从标准的黑箱范式转变为学习更透明的范式，将使我们能够开发新的和改进的强化学习方法，以更好地利用人类的反馈，并帮助设置学习中涉及的众多超参数。Borealis AI正在寻求应用可视化技术来解决这两个关键问题的帮助。**该项目的目标是调查沉浸式可视化是否可以提高非技术和专家用户对强化学习过程的理解。我们将使用HTC Vive和微软HoloLens探索虚拟现实和增强现实（VR和AR）中的可视化技术，以回答以下问题：1)在沉浸式环境中，代理用于学习的策略有哪些不同的可视化方法，这有助于外行观众理解策略？2)我们如何在沉浸式环境中可视化算法的内部数据（例如，神经网络），以展示知识在学习过程中是如何演变的？**

Borealis AI is the Royal Bank of Canada (RBC) Institute for Research, pursuing fundamental and applied research in reinforcement learning (RL), deep learning, adversarial machine learning, and other related areas. In addition to applying state of the art methods, they develop new RL techniques specifically designed to address these real-world challenges. One of the fundamental improvements to the framework is by incorporating human knowledge - leveraging a user's abilities can help the system learn faster and better across a wide range of tasks. The long-term goal of this research is to learn from both explicit and implicit user feedback to the system. In order to make progress towards this goal, there are two challenges that must be overcome. First, the non-technical trainer must be able to understand an agent's policy, or way of acting. We hypothesize that the better the trainer understands what the agent does and does not know, the better she will be able to provide assistance. Second, the Borealis AI RL experts must better understand how a trainer's inputs are affecting the algorithm. We hypothesize that moving from the standard black box paradigm to one in which learning is more transparent will allow us to develop new and improved RL methods to better leverage human feedback, as well as to help set the numerous hyper-parameters involved in learning. Borealis AI is looking for help to apply visualization techniques to attack these two critical problems. **The project's goal is to investigate whether immersive visualizations can improve understanding of the RL process for both non-technical and expert users. We will explore visualization techniques in Virtual and Augmented Reality (VR and AR) using the HTC Vive and the Microsoft HoloLens to answer the following questions: 1) What are different ways to visualize the policies that agents use for learning in an immersive environment, which helps a lay audience understand the policies? 2) How can we visualize the algorithm's internal data (e.g., a neural network) in an immersive environment to demonstrate how the knowledge is evolving during the learning process?**