FW-HTF-RM: Collaborative Research: Assistive Intelligence for Cooperative Robot and Inspector Survey of Infrastructure Systems (AI-CRISIS)

FW-HTF-RM：协作研究：协作机器人辅助智能和基础设施系统检查员调查 (AI-CRISIS)

• 批准号: 2025929
• 负责人: Zhaozheng Yin
• 金额: $ 40.2万
• 依托单位: SUNY at Stony Brook
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2025929&HistoricalAwards=false
• 关键词: FW; HTF; RM; Collaborative; Research

The U.S. civil infrastructure faces the prospect of rapid future deterioration. For example, 39% of the over 600,000 highway bridges in the U.S. have exceeded their design life of 50 years. To ensure the safety and welfare of citizens, state Departments of Transportation are required to inspect bridges every two years. To make inspection and reporting objective and consistent, robots such as unmanned aerial vehicles have recently been introduced to perform autonomous surveys of bridges. These robots, equipped with cameras, will rapidly provide a large set of survey data that can be used to aid inspectors in assessing the condition of bridges. To transform these multidisciplinary topics into a new integrated bridge inspection capability, this multi-university team will develop and implement a cooperative robot-inspector system with assistive intelligence (AI) in order to make the future bridge inspection significantly faster, cheaper, safer, and more consistent. A robotic platform equipped with infrared cameras and a central processing unit with intelligent algorithms will operate in both flying and crawling modes, travel in proximity to various parts/elements of a bridge, and collect high-fidelity images of the entire bridge. Inspectors will be instrumented and monitored with a suite of wearable sensors to enable optimal robot-inspector cooperation during each bridge inspection. AI algorithms will be developed in modules to analyze the big data from human sensors and cameras in support of the pre-inspection workforce training, during-inspection element defect detection, and post-inspection condition evaluation of bridges. The robot-inspector-AI system will be integrated and validated at six bridge sites in collaboration with stakeholders.This multidisciplinary project aims to explore and develop the scientific knowledge and underling methods of AI for a cooperative inspector-robot survey of bridges. The scope of work includes: (1) to develop an optimal screening-to-probing inspection strategy using fast and accurate detection of both surface and internal defects in reinforced concrete slabs (representative bridge elements) from active thermal and visible light imaging; (2) to create a causal model of inspectors’ task performance to support an optimal job design, effective training, and onsite operation; (3) to develop adaptive AI algorithms with human inspectors in the loop for a rapid adaptation to future work contexts, and efficient and reliable data analyses; and (4) to enable a spatial correlation among the elements of an entire bridge with a maximum use of domain expertise, and thus improve the effectiveness and applicability of adaptive AI. The methods and technologies developed in this study will be transferred into the hands of stakeholders, researchers, policy makers, and end users (bridge inspectors) through publications, presentations, field demonstrations, and training. In particular, the world’s first bridge benchmark dataset of videos, labeled objects, and metadata established from real-world bridges, upon approval by stakeholders, will be shared with the international community via an AI-enabled bridge inspection competition.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

美国的民用基础设施面临着未来迅速恶化的前景。例如，在美国60多万座公路桥梁中，39%的桥梁已经超过了50年的设计寿命。为了确保公民的安全和福利，国家交通部门被要求每两年对桥梁进行一次检查。为了使检查和报告具有客观性和一致性，最近引入了无人驾驶飞行器等机器人来对桥梁进行自主测量。这些配备了摄像头的机器人将迅速提供大量调查数据，可用于帮助检查人员评估桥梁状况。为了将这些多学科的主题转化为新的综合桥梁检测能力，这个多大学团队将开发和实施一个具有辅助智能(AI)的协作机器人-检查员系统，以使未来的桥梁检测显著更快、更便宜、更安全、更一致。一个配备了红外摄像头和带有智能算法的中央处理器的机器人平台将在飞行和爬行模式下运行，在桥梁的各个部件/部件附近移动，并收集整个桥梁的高保真图像。检查员将使用一套可穿戴传感器进行仪表和监控，以便在每次桥梁检查期间实现机器人与检查员的最佳合作。AI算法将以模块的形式开发，以分析来自人类传感器和摄像头的大数据，以支持检测前的劳动力培训、检测中的构件缺陷检测和检测后的桥梁状况评估。机器人-检查员-人工智能系统将与利益相关者合作，在六个桥梁站点进行集成和验证。这个多学科项目旨在探索和发展人工智能的科学知识和底层方法，以实现桥梁检查员-机器人的合作调查。工作范围包括：(1)利用主动热成像和可见光成像，快速准确地检测钢筋混凝土板(有代表性的桥梁构件)的表面和内部缺陷，从而制定最佳的筛选-探测检测策略；(2)创建检查员任务表现的因果模型，以支持最佳工作设计、有效的培训和现场操作；(3)开发具有人类检查员参与的自适应人工智能算法，以便快速适应未来的工作环境，并进行高效和可靠的数据分析；(4)最大限度地利用领域专业知识，实现整个桥梁元素之间的空间相关性，从而提高自适应人工智能的有效性和适用性。本研究中开发的方法和技术将通过出版物、演示、现场演示和培训转移到利益相关者、研究人员、政策制定者和最终用户(桥梁检查员)手中。特别是，世界上第一个桥梁基准数据集，包括视频、标记对象和从现实世界桥梁建立的元数据，经利益相关者批准后，将通过人工智能支持的桥梁检查比赛与国际社会共享。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Global Memory and Local Continuity for Video Object Detection

• DOI: 10.1109/tmm.2022.3164253
• 发表时间: 2023
• 期刊: IEEE Transactions on Multimedia
• 影响因子: 7.3
• 作者: Liang Han;Zhaozheng Yin

A Dynamic Spatial-Temporal Attention Network for Early Anticipation of Traffic Accidents

• DOI: 10.1109/tits.2022.3155613
• 发表时间: 2022-03-09
• 期刊: IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS
• 影响因子: 8.5
• 作者: Karim, Muhammad Monjurul;Li, Yu;Yin, Zhaozheng
• 通讯作者: Yin, Zhaozheng

An Attention-Guided Multistream Feature Fusion Network for Early Localization of Risky Traffic Agents in Driving Videos

• DOI: 10.1109/tiv.2023.3275543
• 发表时间: 2024-01
• 期刊: IEEE Transactions on Intelligent Vehicles
• 影响因子: 8.2
• 作者: Muhammad Monjurul Karim;Zhaozheng Yin;Ruwen Qin

A semi-supervised self-training method to develop assistive intelligence for segmenting multiclass bridge elements from inspection videos

• DOI: 10.1177/14759217211010422
• 发表时间: 2021-05
• 期刊: Structural Health Monitoring
• 作者: Muhammad Monjurul Karim;Ruwen Qin;Genda Chen;Zhaozheng Yin

Collaborative Foreground, Background, and Action Modeling Network for Weakly Supervised Temporal Action Localization

• DOI: 10.1109/tcsvt.2023.3272891
• 发表时间: 2023-11
• 期刊: IEEE Transactions on Circuits and Systems for Video Technology
• 影响因子: 8.4
• 作者: Md Moniruzzaman;Zhaozheng Yin