CAPS: Comprehensible Abstract Policy Summaries for Explaining Reinforcement Learning Agents
CAPS: Comprehensible Abstract Policy Summaries for Explaining Reinforcement Learning Agents
复制标题
DOI：
10.5555/3535850.3535950
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Joe McCalmon;Thai Le;Sarra M. Alqahtani;Dongwon Lee
中科院分区：
文献类型：
--
作者：
Joe McCalmon;Thai Le;Sarra M. Alqahtani;Dongwon Lee
随着强化学习(RL)的不断改进和在人类环境中的应用，向最终用户解释RL代理的学习行为变得更加重要。解释代理策略背后的推理的策略称为策略级解释，可以导致对任务和代理行为的重要洞察。根据这一研究思路，在本工作中，我们提出了一种新的方法，称为CAPS，它以自然语言描述的有向图的形式总结了代理的策略。使用基于决策树的聚类方法将任务的状态空间抽象为更少的浓缩状态，使最终用户更容易理解策略图。这种抽象允许用户控制策略图的大小，以在可理解性和准确性之间实现他们所需的平衡。此外，我们还开发了一种启发式优化方法来寻找最可解释的图策略，并将其呈现给用户。最后，我们使用用户定义的谓词来丰富抽象状态的语义。我们使用确定性和随机策略在5个RL任务上测试了我们的方法，并表明我们的方法是：(1)与用于训练策略的算法无关，(2)精度与现有基线相当，解释能力优于现有基线。特别是，当提供我们的解释图表时，最终用户能够在80%的时间内准确地解释训练有素的RL代理的策略，而在提供次佳基准时，这一比例为10%。我们提供我们的代码和数据集，以确保研究的重复性
As reinforcement learning (RL) continues to improve and be applied in situations alongside humans, the need to explain the learned behaviors of RL agents to end-users becomes more important. Strategies for explaining the reasoning behind an agent’s policy, called policy-level explanations , can lead to important insights about both the task and the agent’s behaviors. Following this line of research, in this work, we propose a novel approach, named as CAPS , that summarizes an agent’s policy in the form of a directed graph with natural language descriptions. A decision tree based clustering method is utilized to abstract the state space of the task into fewer, condensed states which makes the policy graphs more digestible to end-users. This abstraction allows the users to control the size of the policy graph to achieve their desired balance between comprehensi-bility and accuracy. In addition, we develop a heuristic optimization method to find the most explainable graph policy and present it to the users. Finally, we use the user-defined predicates to enrich the abstract states with semantic meaning. We test our approach on 5 RL tasks, using both deterministic and stochastic policies, and show that our method is: (1) agnostic to the algorithms used to train the policies, and (2) comparable in accuracy and superior in explanation capabilities to existing baselines. Especially, when provided with our explanation graph, end-users are able to accurately interpret policies of trained RL agents 80% of the time, compared to 10% when provided with the next best baseline. We make our code and datasets available to ensure the reproducibility of our research