Towards interactive explanatory reinforcement learning for aligned and trustworthy agents

深度强化学习(RL)是解决复杂控制问题的有效方法，但缺乏透明度阻碍了信任、理解和安全性验证。如果RL要为未来的自治系统提供动力，那么开发用于“打开黑匣子”的可解释性工具必须成为一门严谨的科学。我的研究采用可解释人工智能(XAI)方法来分析深层RL代理的行为。在我工作的第一年，我使用决策树“克隆”了多智能体交通控制器，揭示了影响其输出的潜在因素。自那以后，我开发了一个新的树模型，用于联合表示月球着陆器的政策、价值函数和时间动态，便于交互可视化和查询回答。虽然控制的闭环性质使得解释比监督学习更复杂，但通过利用人类的能力来采用Dennett对代理的有意立场，最终结果可能会更直观。XAI研究人员的工作应该总是有一个目标用户。在接下来的几个月里，我打算把重点放在一个特定的用户身上：RL代理本身的设计师，他负责定义模型参数，并至关重要地指定驱动学习的奖励功能。我计划使用XAI来促进交互式RL，其中对奖励函数和学习动态的更深层次的因果理解使得训练、调整和奖励修改能够更有原则地迭代，取代今天的试错方法。XAI的这一应用解决了对齐的哲学问题，该问题强调了在功能日益强大的通用优化器的背景下，正确的目标规范是至关重要的。这项研究属于EPSRC人工智能技术研究领域，也与人机交互有关。

Deep reinforcement learning (RL) is a compelling solution to complex control problems, but a lack of transparency hampers trust, understanding and safety validation. If RL is to power future autonomous systems, the development of interpretability tools to "open the black box" must become a rigorous science. My research adapts explainable AI (XAI) methods to analyse the behaviour of deep RL agents. In my first year I have used decision trees to "clone" multiagent traffic controllers, revealing the latent factors influencing their outputs. I have since developed a novel tree model for jointly representing the policy, value function and temporal dynamics of a lunar lander, facilitating interactive visualisation and query-answering. While the closed-loop nature of control makes explanation more complex than in supervised learning, the end result may yet be more intuitive, by leveraging human's capacity to adopt Dennett's intentional stance with respect to agents.XAI researchers should always have a target user for their work. In the coming months I intend to focus on one particular user: the designer of the RL agent itself, who is responsible for defining model parameters, and crucially specifying the reward function that drives learning. I plan to use XAI to facilitate interactive RL, in which deeper causal understanding of reward functions and learning dynamics enables the more principled iteration of training, tuning and reward modification, replacing today's trial-and-error approach. This application of XAI addresses the philosophical problem of alignment, which stresses the paramount importance of correct goal specification in the context of increasingly powerful generic optimisers. This research falls within the EPSRC Artificial Intelligence Technologies research area, and also has connections to Human-computer Interaction.