CAREER: Securing Deep Reinforcement Learning

与许多其他深度学习技术一样，深度强化学习很容易受到对抗性攻击。在强化学习中，对抗性攻击操纵强化学习代理的感官观察，使其陷入混乱。最近，研究表明，对抗性攻击可能更加实用。新类型的攻击不再隐含地假设攻击者有完全的控制权来影响智能体的感官系统，而是呈现一个对抗的智能体来操纵目标智能体的环境，从而触发它以一种不希望的方式做出反应。与改变感官观察的那种攻击相比，新的攻击更难以抵消。首先，通常用于增强其他深度学习技术的方法（例如，对抗性训练）不再适合深度强化学习。其次，给定一个强化学习代理，很少有技术方法来仔细检查代理并揭示其缺陷。这个项目打算通过整合和扩展一系列技术方法来解决这两个重要的问题，这些技术方法用于可解释的人工智能、对抗性训练，以及与程序合成相结合的正式验证。基本的想法是首先学习一个由可解释的AI告知的对抗性代理。使用这个学习代理，我们揭示目标代理的弱点，并相应地对抗性训练它们。通过鲁棒性检查，我们评估了增强代理。如果强化代理未能通过对抗性检查，我们就会求助于形式验证和程序合成技术。使用这个统一的解决方案，强化学习模型开发人员可以识别强化学习代理的策略缺陷，并有效地弥补它们的弱点。该项目将提供一套技术解决方案，以审查和增强深度强化学习。如果成功，该项目将显著推进人工智能安全领域（对抗性训练和对抗性政策学习），并为机器学习领域（可解释的人工智能和经过验证的人工智能）做出贡献。此外，该项目具有显著提高强化学习应用安全性的潜力。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Like many other deep learning techniques, deep reinforcement learning is vulnerable to adversarial attacks. In reinforcement learning, an adversarial attack manipulates a reinforcement learning agent's sensory observation, flummoxing it. Recently, research has demonstrated that an adversarial attack could be even more practical. Instead of implicitly assuming an attacker has the full control to influence an agent's sensory system, the new type of attack presents an adversarial agent to manipulate the target agent's environment and thus trigger it to react in an undesired fashion. Compared with the kind of attack that alters the sensory observation, the new attack is more difficult to counteract. First, the methods (e.g., adversarial training) commonly used for robustifying other deep learning techniques are no longer suitable for deep reinforcement learning. Second, given a reinforcement learning agent, there are few technical approaches to scrutinizing the agent and unveiling its flaws. This project intends to address these two significant problems by integrating and expanding upon a series of technical approaches used in explainable AI, adversarial training, and formal verification in conjunction with program synthesis. The basic idea is first to learn an adversarial agent informed by explainable AI. Using this learned agent, we then unveil the weakness of target agents and adversarially train them accordingly. Through a robustness check, we evaluate the enhanced agents. If a strengthened agent fails the adversary-resistance check, we fall back on formal verification and program synthesis techniques. Using this unified solution, reinforcement learning model developers could identify the policy flaws of reinforcement learning agents and effectively remediate their weaknesses. This project will provide a stack of technical solutions to scrutinizing and robustifying deep reinforcement learning. If successful, the project will significantly advance the field of AI security (for adversarial training and adversarial policy learning) and contribute to the field of machine learning (for explainable AI and verified AI). Besides, this project has the potential to improve the security of reinforcement learning applications significantly.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.