Towards interactive explanatory reinforcement learning for aligned and trustworthy agents

为一致且值得信赖的代理提供交互式解释性强化学习

• 批准号: 2314554
• 金额: --
• 依托单位: University of Bristol
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2019
• 资助国家: 英国
• 起止时间: 2019 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2314554
• 关键词: Towards; interactive; explanatory; reinforcement; learning

Deep reinforcement learning (RL) is a compelling solution to complex control problems, but a lack of transparency hampers trust, understanding and safety validation. If RL is to power future autonomous systems, the development of interpretability tools to "open the black box" must become a rigorous science. My research adapts explainable AI (XAI) methods to analyse the behaviour of deep RL agents. In my first year I have used decision trees to "clone" multiagent traffic controllers, revealing the latent factors influencing their outputs. I have since developed a novel tree model for jointly representing the policy, value function and temporal dynamics of a lunar lander, facilitating interactive visualisation and query-answering. While the closed-loop nature of control makes explanation more complex than in supervised learning, the end result may yet be more intuitive, by leveraging human's capacity to adopt Dennett's intentional stance with respect to agents.XAI researchers should always have a target user for their work. In the coming months I intend to focus on one particular user: the designer of the RL agent itself, who is responsible for defining model parameters, and crucially specifying the reward function that drives learning. I plan to use XAI to facilitate interactive RL, in which deeper causal understanding of reward functions and learning dynamics enables the more principled iteration of training, tuning and reward modification, replacing today's trial-and-error approach. This application of XAI addresses the philosophical problem of alignment, which stresses the paramount importance of correct goal specification in the context of increasingly powerful generic optimisers. This research falls within the EPSRC Artificial Intelligence Technologies research area, and also has connections to Human-computer Interaction.

深度强化学习（RL）是解决复杂控制问题的一种令人信服的解决方案，但缺乏透明度会阻碍信任、理解和安全验证。如果RL要为未来的自主系统提供动力，那么开发可解释性工具来“打开黑盒子”必须成为一门严格的科学。我的研究采用可解释AI（XAI）方法来分析深度RL代理的行为。在我的第一年，我已经使用决策树“克隆”多智能体交通控制器，揭示潜在的因素影响他们的输出。从那以后，我开发了一种新的树模型，用于联合表示月球着陆器的政策，价值函数和时间动态，促进交互式可视化和问答。虽然控制的闭环性质使得解释比监督学习更复杂，但最终结果可能更直观，通过利用人类的能力来采用Dennett关于代理的意向立场。XAI研究人员应该始终有一个目标用户。在接下来的几个月里，我打算专注于一个特定的用户：RL代理本身的设计者，他负责定义模型参数，并重要地指定驱动学习的奖励函数。我计划使用XAI来促进交互式RL，其中对奖励函数和学习动态的更深入的因果理解使得训练，调整和奖励修改的迭代更加原则化，取代今天的试错方法。XAI的这种应用解决了对齐的哲学问题，它强调了在日益强大的通用优化器的背景下正确的目标规范的至关重要性。这项研究属于EPSRC人工智能技术研究领域的福尔斯，也与人机交互有关。