CAREER: Probabilistic Risk Evaluation for Safety-Critical Intelligent Autonomy

在人工智能(AI)最新进展的推动下，创新表现出了与人类竞争的表现。然而，随着研究扩展到自动驾驶汽车和医疗保健等安全关键应用，它们的安全性问题成为从理论过渡到实践的瓶颈。安全关键的自治性在大规模部署之前必须经过严格的评估。它们的独特之处在于，故障可能会导致严重后果，因此需要极低的故障率。这意味着在自然条件下的测试结果是极其不平衡的--失败的案例很少。这种稀缺性，再加上复杂的人工智能结构，给设计传统方法无法充分解决的有效评估方法带来了巨大的挑战。这项建议旨在了解在评估安全关键的人工智能自主风险方面的根本挑战，并提出新的理论和实践工具来开发可认证、可实施和高效的评估程序。本研究的具体目的是开发三种类型的人工智能自主性的评估方法，这三种类型的人工智能自主性涵盖了广泛的现实世界应用：深度学习系统、强化学习系统和包含子模块的复杂系统，并用真实自主系统的感知和决策系统来验证它们。这项研究为PI在物理世界安全部署人工智能的长期职业目标奠定了基础，开辟了一个新的交叉领域来开发严格和高效的评估方法，解决了即将到来的大规模人工智能自主部署的紧迫社会关切，并通过各级教育培养了一支多样化的、具有全球竞争力的劳动力队伍。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Innovations driven by recent progress in artificial intelligence (AI) have demonstrated human-competitive performance. However, as research expands to safety-critical applications, such as autonomous vehicles and healthcare treatment, the question of their safety becomes a bottleneck for the transition from theories to practice. Safety-critical autonomy must go through a rigorous evaluation before massive deployment. They are unique in the sense that failures may cause serious consequences, thus requiring an extremely low failure rate. This means that test results under naturalistic conditions are extremely imbalanced - with the failure cases being rare. The rarity, together with the complex AI structures, poses a huge challenge to design effective evaluation methods that cannot be adequately addressed by conventional methods. This proposal aims to understand the fundamental challenges in assessing the risk of safety-critical AI autonomy and puts forward new theories and practical tools to develop certifiable, implementable, and efficient evaluation procedures. The specific aims of this research are to develop evaluation methods for three types of AI autonomy that cover a broad array of real-world applications: deep learning systems, reinforcement learning systems, and sophisticated systems comprising sub-modules, and validate them with the sensing and decision-making systems of real-world autonomous systems. This research lays the foundation for the PI’s long-term career goal to safely deploy AI in the physical world, opens up a new cross-cutting area to develop rigorous and efficient evaluation methods, addresses the urgent societal concern with the upcoming massive deployment of AI autonomy, and train a diverse, globally competitive workforce through education at all levels.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.