Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles
Reliability analysis of artificial intelligence systems using recurrent events data from autonomous vehicles
复制标题
DOI：
10.1111/rssc.12564
复制
发表时间：
2021-02
期刊：
影响因子：
--
通讯作者：
Yili Hong;Jie Min;Caleb King;W. Meeker
中科院分区：
文献类型：
--
作者：
Yili Hong;Jie Min;Caleb King;W. Meeker
人工智能（AI）系统已经变得越来越普遍，并且这一趋势将持续下去。人工智能系统的例子包括自动驾驶汽车 (AV)、计算机视觉、自然语言处理和人工智能医疗专家。为了安全有效地部署人工智能系统，需要评估此类系统的可靠性。传统上，可靠性评估是基于可靠性测试数据以及随后的统计建模和分析。然而，人工智能系统的可靠性数据的可用性是有限的，因为这些数据通常是敏感的和专有的。加州机动车辆管理局 (DMV) 负责监督和管理自动驾驶汽车测试计划，许多自动驾驶汽车制造商正在该计划中进行自动驾驶汽车道路测试。参与该计划的制造商必须向加州 DMV 报告经常发生的脱离事件。该信息正在向公众公开。在本文中，我们使用经常性脱离事件作为自动驾驶人工智能系统可靠性的表示，并提出了一个统计框架来建模和分析自动驾驶驾驶测试中的经常性事件数据。我们使用软件可靠性中的传统参数模型，并提出一种基于单调样条的新非参数模型来描述事件过程并估计事件过程的累积基线强度函数。我们开发推理程序来选择最佳模型、量化不确定性并测试事件过程中的异质性。然后，我们分析了四家自动驾驶汽车制造商的重复事件数据，并对自动驾驶汽车中人工智能系统的可靠性做出了推断。我们还描述了如何应用所提出的分析来评估其他人工智能系统的可靠性。本文有在线补充材料。
Artificial intelligence (AI) systems have become increasingly common and the trend will continue. Examples of AI systems include autonomous vehicles (AV), computer vision, natural language processing and AI medical experts. To allow for safe and effective deployment of AI systems, the reliability of such systems needs to be assessed. Traditionally, reliability assessment is based on reliability test data and the subsequent statistical modelling and analysis. The availability of reliability data for AI systems, however, is limited because such data are typically sensitive and proprietary. The California Department of Motor Vehicles (DMV) oversees and regulates an AV testing program, in which many AV manufacturers are conducting AV road tests. Manufacturers participating in the program are required to report recurrent disengagement events to California DMV. This information is being made available to the public. In this paper, we use recurrent disengagement events as a representation of the reliability of the AI system in AV, and propose a statistical framework for modelling and analysing the recurrent events data from AV driving tests. We use traditional parametric models in software reliability and propose a new non‐parametric model based on monotonic splines to describe the event process and to estimate the cumulative baseline intensity function of the event process. We develop inference procedures for selecting the best models, quantifying uncertainty and testing heterogeneity in the event process. We then analyse the recurrent events data from four AV manufacturers, and make inferences on the reliability of the AI systems in AV. We also describe how the proposed analysis can be applied to assess the reliability of other AI systems. This paper has online supplementary materials.