Towards Driving-Oriented Metric for Lane Detection Models
Towards Driving-Oriented Metric for Lane Detection Models
复制标题
DOI：
10.1109/cvpr52688.2022.01664
复制
发表时间：
2022-03
期刊：
影响因子：
--
通讯作者：
Takami Sato;Qi Alfred Chen
中科院分区：
文献类型：
--
作者：
Takami Sato;Qi Alfred Chen
在2017年tussimple车道检测挑战赛之后，其数据集以及基于准确率和F1分数的评估已经成为衡量车道检测方法性能的事实上的标准。虽然它们在提高车道检测方法的性能方面发挥了重要作用，但该评估方法在下游任务中的有效性尚未得到充分研究。在本研究中，我们设计了2个新的以驾驶为导向的车道检测度量：端到端横向偏差度量（E2E-LD）是根据自动驾驶的需求直接制定的，这是车道检测的核心下游任务；逐帧模拟横向偏差度量（PSLD）是E2E-LD的轻量级替代度量。为了评估指标的有效性，我们在TuSimple数据集和我们新构建的数据集Comma2k19-LD上进行了4种主要车道检测方法的大规模实证研究。我们的研究结果表明，传统指标与E2E-LD具有很强的负相关性（≤-0.55），这意味着最近一些纯粹针对传统指标的改进可能不会导致自动驾驶的有意义的改进，反而可能会因为过度拟合传统指标而使情况变得更糟。由于自动驾驶是一个安全/安全关键系统，对鲁棒性的低估阻碍了实际车道检测模型的良好发展。我们希望我们的研究将有助于社区实现更多的下游任务感知评估车道检测。
After the 2017 TuSimple Lane Detection Challenge, its dataset and evaluation based on accuracy and F1 score have become the de facto standard to measure the performance of lane detection methods. While they have played a major role in improving the performance of lane detection methods, the validity of this evaluation method in down-stream tasks has not been adequately researched. In this study, we design 2 new driving-oriented metrics for lane detection: End-to-End Lateral Deviation metric (E2E-LD) is directly formulated based on the requirements of autonomous driving, a core downstream task of lane detection; Per-frame Simulated Lateral Deviation metric (PSLD) is a lightweight surrogate metric of E2E-LD. To evaluate the validity of the metrics, we conduct a large-scale empirical study with 4 major types of lane detection approaches on the TuSimple dataset and our newly constructed dataset Comma2k19-LD. Our results show that the conventional metrics have strongly negative correlations (≤-0.55) with E2E-LD, meaning that some recent improvements purely targeting the conventional metrics may not have led to meaningful improvements in autonomous driving, but rather may actually have made it worse by over-fitting to the conventional metrics. As autonomous driving is a security/safety-critical system, the underestimation of robustness hinders the sound development of practical lane detection models. We hope that our study will help the community achieve more downstream task-aware evaluations for lane detection.