Towards Safe AI: Sandboxing DNNs-Based Controllers in Stochastic Games
Towards Safe AI: Sandboxing DNNs-Based Controllers in Stochastic Games
复制标题
DOI：
10.1609/aaai.v37i12.26789
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Bingzhuo Zhong;H. Cao;Majid Zamani;M. Caccamo
中科院分区：
文献类型：
--
作者：
Bingzhuo Zhong;H. Cao;Majid Zamani;M. Caccamo
如今，基于人工智能的技术，如深度神经网络(DNN)，被广泛部署在自治系统中，以满足复杂的任务要求(例如，机器人中的运动规划)。然而，基于DNN的控制器通常非常复杂，很难正式验证其正确性，这可能会给安全关键型自治系统带来严重风险。在本文中，我们提出了一种所谓的安全监控体系结构的构建方案，用于沙箱基于DNNS的控制器。特别是，我们考虑了在随机博弈框架下的构造，以提供对噪声和扰动具有鲁棒性的系统级安全保证。管理程序用于检查基于DNNS的控制器提供的控制输入，并决定是否接受它们。同时，安全顾问正在并行运行，以便在基于DNN的控制器被拒绝时提供后备控制输入。我们在一个采用未经验证的基于DNN控制器的四旋翼上演示了所提出的方法。
Nowadays, AI-based techniques, such as deep neural networks (DNNs), are widely deployed in autonomous systems for complex mission requirements (e.g., motion planning in robotics). However, DNNs-based controllers are typically very complex, and it is very hard to formally verify their correctness, potentially causing severe risks for safety-critical autonomous systems. In this paper, we propose a construction scheme for a so-called Safe-visor architecture to sandbox DNNs-based controllers. Particularly, we consider the construction under a stochastic game framework to provide a system-level safety guarantee which is robust to noises and disturbances. A supervisor is built to check the control inputs provided by a DNNs-based controller and decide whether to accept them. Meanwhile, a safety advisor is running in parallel to provide fallback control inputs in case the DNN-based controller is rejected. We demonstrate the proposed approaches on a quadrotor employing an unverified DNNs-based controller.