Evaluation Metrics for Object Detection for Autonomous Systems
Evaluation Metrics for Object Detection for Autonomous Systems
复制标题
DOI：
10.48550/arxiv.2210.10298
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Apurva Badithela;T. Wongpiromsarn;R. Murray
中科院分区：
文献类型：
--
作者：
Apurva Badithela;T. Wongpiromsarn;R. Murray
本文研究了基于学习的目标检测模型的评估，并结合定义在自治系统及其环境的抽象模型上的形式化规范的模型检测。特别地，我们定义了两个用于评估对象检测的度量--\emph{命题标签}和\emph{类标签}混淆矩阵，并结合这些度量来计算系统级安全需求的满足概率。虽然混淆矩阵对于分类和目标检测模型的比较评估是有效的，但我们的框架填补了两个关键空白。首先，我们将目标检测的性能与在下游高层规划任务上定义的形式需求相关联。特别是，我们提供的实证结果表明，就整体系统的形式要求而言，选择一个好的目标检测算法在很大程度上取决于下游的规划和控制设计。其次，与传统的混淆矩阵不同，我们的度量考虑了自我与被检测对象之间的距离在性能上的变化。通过计算线性时态逻辑(LTL)形式化的安全需求的满足概率，我们以汽车-行人为例演示了该框架。
This paper studies the evaluation of learning-based object detection models in conjunction with model-checking of formal specifications defined on an abstract model of an autonomous system and its environment. In particular, we define two metrics -- \emph{proposition-labeled} and \emph{class-labeled} confusion matrices -- for evaluating object detection, and we incorporate these metrics to compute the satisfaction probability of system-level safety requirements. While confusion matrices have been effective for comparative evaluation of classification and object detection models, our framework fills two key gaps. First, we relate the performance of object detection to formal requirements defined over downstream high-level planning tasks. In particular, we provide empirical results that show that the choice of a good object detection algorithm, with respect to formal requirements on the overall system, significantly depends on the downstream planning and control design. Secondly, unlike the traditional confusion matrix, our metrics account for variations in performance with respect to the distance between the ego and the object being detected. We demonstrate this framework on a car-pedestrian example by computing the satisfaction probabilities for safety requirements formalized in Linear Temporal Logic (LTL).