Collaborative Research: CPS: Medium: Spatio-Temporal Logics for Analyzing and Querying Perception Systems

合作研究：CPS：媒介：用于分析和查询感知系统的时空逻辑

• 批准号: 2038666
• 负责人: Yezhou Yang
• 金额: $ 79.99万
• 依托单位: Arizona State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-01-01 至 2024-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2038666&HistoricalAwards=false
• 关键词: Collaborative; Research; CPS; Medium; Spatio

The goals of Automated Driving Systems (ADS) and Advanced Driver Assistance Systems (ADAS) include reduction in accidental deaths, enhanced mobility for differently abled people, and an overall improvement in the quality of life for the general public. Such systems typically operate in open and highly uncertain environments for which robust perception systems are essential. However, despite the tremendous theoretical and experimental progress in computer vision, machine learning, and sensor fusion, the form and conditions under which guarantees should be provided for perception components is still unclear. The state-of-the-art is to perform scenario-based evaluation of data against ground truth values, but this has only limited impact. The lack of formal metrics to analyze the quality of perception systems has already led to several catastrophic incidents and a plateau in ADS/ADAS development. This project develops formal languages for specifying and evaluating the quality and robustness of perception sub-systems within ADS and ADAS applications. To enable broader dissemination of this technology, the project develops graduate and undergraduate curricula to train engineers in the use of such methods, and new educational modules to explain the challenges in developing safe and robust ADS for outreach and public engagement activities. To broaden participation in computing, the investigators target the inclusion of undergraduate women in research and development phases through summer internships.The formal language developed in this project is based on a new spatio-temporal logic pioneered by the investigators. This logic allows one to simultaneously perform temporal reasoning about streaming perception data, and spatially reason about objects both within a single frame of the data and across frames. The project also develops quantitative semantics for this logic, which provides the user with quantifiable quality metrics for perception sub-systems. These semantics enable comparisons between different perception systems and architectures. Crucially, the formal language facilitates the process of abstracting away implementation details, which in turn allows system designers and regulators to specify assumptions and guarantees for system performance at a higher-level of abstraction. An interesting benefit of this formal language is that it enables querying of databases with perception data for specific driving scenarios without the need for the highly manual process of creating ground truth annotations. Such a formal language currently does not exist, and this is a huge impediment to building a thriving marketplace for perception components used in safety-critical systems. This framework sets the foundation for a requirements language between suppliers of perception components and automotive companies. The open source and publicly available software tools developed in this project will assist with testing of perception systems by engineers and governmental agencies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

自动驾驶系统（ADS）和高级驾驶员辅助系统（ADAS）的目标包括减少意外死亡，增强不同能力人群的移动性，以及全面提高公众的生活质量。这种系统通常在开放和高度不确定的环境中运行，鲁棒的感知系统是必不可少的。然而，尽管在计算机视觉、机器学习和传感器融合方面取得了巨大的理论和实验进展，但为感知组件提供保证的形式和条件仍然不清楚。最先进的方法是根据地面实况值对数据进行基于MIMO的评估，但这只会产生有限的影响。缺乏正式的指标来分析感知系统的质量已经导致了几次灾难性的事件和ADS/ADAS开发的停滞。该项目开发用于指定和评估ADS和ADAS应用中感知子系统的质量和鲁棒性的正式语言。 为了能够更广泛地传播这种技术，该项目开发了研究生和本科生课程，以培训工程师使用这种方法，并开发了新的教育模块，以解释在开发安全和强大的ADS用于外联和公众参与活动方面的挑战。为了扩大对计算的参与，调查人员的目标是通过暑期实习将本科女生纳入研究和开发阶段。该项目开发的正式语言基于调查人员开创的新时空逻辑。该逻辑允许同时执行关于流式感知数据的时间推理，以及关于数据的单个帧内和跨帧的对象的空间推理。 该项目还开发了定量语义的逻辑，它为用户提供了可量化的质量指标的感知子系统。这些语义使不同的感知系统和架构之间的比较。最重要的是，形式化语言促进了抽象实现细节的过程，这反过来又允许系统设计者和监管者在更高的抽象级别上指定对系统性能的假设和保证。这种形式化语言的一个有趣的好处是，它可以使用特定驾驶场景的感知数据查询数据库，而无需创建地面实况注释的高度手动过程。这种形式化的语言目前还不存在，这对于为安全关键系统中使用的感知组件建立繁荣的市场是一个巨大的障碍。该框架为感知组件供应商和汽车公司之间的需求语言奠定了基础。该项目开发的开源和公开可用的软件工具将有助于工程师和政府机构测试感知系统。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Targeted Attack on Deep RL-based Autonomous Driving with Learned Visual Patterns

• DOI: 10.1109/icra46639.2022.9811574
• 发表时间: 2021-09
• 期刊: 2022 International Conference on Robotics and Automation (ICRA)
• 作者: Prasanth Buddareddygari;Travis Zhang;Yezhou Yang;Yi Ren-

PerceMon: Online Monitoring for Perception Systems

• DOI: 10.1007/978-3-030-88494-9_18
• 发表时间: 2021-08
• 期刊: ArXiv
• 作者: Anand Balakrishnan;Jyotirmoy V. Deshmukh;Bardh Hoxha;Tomoya Yamaguchi;Georgios Fainekos

Covariate Shift Detection via Domain Interpolation Sensitivity

• 发表时间: 2022
• 作者: Tejas Gokhale;Joshua Forster Feinglass

Towards Addressing the Misalignment of Object Proposal Evaluation for Vision-Language Tasks via Semantic Grounding

• DOI: 10.1109/wacv57701.2024.00434
• 发表时间: 2023-09
• 期刊: 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
• 作者: Joshua Forster Feinglass;Yezhou Yang

PyFoReL: A Domain-Specific Language for Formal Requirements in Temporal Logic

PyFoReL：用于时态逻辑中形式要求的特定于领域的语言

• DOI: 10.1109/re54965.2022.00037
• 发表时间: 2022
• 期刊: 2022 IEEE 30th International Requirements Engineering Conference (RE
• 作者: Anderson, Jacob;Hekmatnejad, Mohammad;Fainekos, Georgios
• 通讯作者: Fainekos, Georgios