AR-integrated intelligent visual inspection system for health monitoring of constructed facilities

建筑物、桥梁、管道等已建成设施的老化是一个世界性的严重问题，因为它影响到设施的安全。在加拿大，许多已建成的设施，特别是基础设施，都位于人口稀少或土著地区，开发能够大幅减少人力需求、提高检查效率和效果的自动化智能视觉检查系统是必要的。在这方面，本研究计划的目标是开发近实时(NRT)快速缺陷识别算法，并利用增强现实(AR)开发一种AR集成的智能视觉检测系统，用于近实时监测已建设施的健康状况。该智能系统将包括两种模式：(1)自动无人机检测模式(M1)；(2)面对面检测模式(M2)。对于M1，GPS制导的无人机将沿着预先规划的路线飞行，自动捕获设施表面检查数据(例如，视频或照片)，并在平板电脑或智能手机上实时显示这些数据，然后在显示的视图上立即叠加识别出的缺陷(即AR效果)。无论何时发现缺陷，都会记录相应的GPS位置。在无人机完成整个计划路线后，它会自动飞回每个记录的位置，以捕获更详细的关于被发现缺陷的数据。对于M2，AR-HMD(头盔显示器)将捕获检测数据并将其传输到笔记本电脑进行快速缺陷识别，然后将识别出的缺陷叠加到AR-HMD上。AR-HMD的交互式手势控制将被纳入开发中。为了实现该计划的目标，制定了四项研究目标：(1)开发针对钢铁和钢筋混凝土(RC)建筑设施的近实时快速缺陷识别算法；(2)针对M1模式，将无人机与iOS/iPadOS和Android系统的平板电脑/智能手机集成；(3)针对M2模式，将AR-HMD及其手势控制与笔记本电脑集成；以及(4)针对这两种模式设计和开发用户友好的AR用户界面。该项目将培训9名高素质人才(HQP)，包括3名博士生、4名MASC学生和2名本科研究助理(URA)。目前，建筑、工程和建筑(AEC)行业的AR应用大多依赖于AR设备上预加载的数据来进行目标或模式识别。该程序将通过以接近实时的方式处理新捕获的数据，将AR应用提升到一个新的水平，这导致了需要开发的快速缺陷识别算法的重要性。同时，拟议的自动无人机检查模式和带有交互式AR-HMD的面对面检查模式都可能彻底改变当前的检查做法，并使加拿大及其他地区的公共和私营工程部门显著受益。

The aging of constructed facilities, such as buildings, bridges, pipelines, etc., is a critical issue worldwide, as it affects the safety of the facilities. In Canada, many constructed facilities, especially infrastructure facilities, are in sparsely populated or indigenous areas, and the development of an automated intelligent visual inspection system that can greatly reduce manpower requirement and enhance inspection efficiency and effectiveness is necessary. In this regard, the goal of this research program is to develop near-real-time (NRT) fast defect recognition algorithms and utilize augmented reality (AR) to develop an AR-integrated intelligent visual inspection system for near-real-time health monitoring of constructed facilities. This intelligent system will comprise two modes: (1) automated drone inspection mode (M1); and (2) in-person inspection mode (M2). For M1, a GPS-guided drone will fly along a pre-planned route to automatically capture facility surface inspection data (e.g., videos or photos), and display them on a tablet or smartphone in real time, followed by immediate superimposing of recognized defects on the displayed view (i.e., AR effects). Whenever a defect is identified, the corresponding GPS position will be recorded. After the drone finishes the entire planned route, it will automatically fly back to each of the recorded positions to capture more detailed data around the spotted defect. For M2, an AR-HMD (head-mounted display) will capture inspection data and transmit them to a laptop for fast defect recognition, followed by superimposing recognized defects on AR-HMD. Interactive gesture control of AR-HMD will be incorporated in the development. To achieve the goal of this program, four research objectives are set: (1) to develop near-real-time fast defect recognition algorithms for steel and reinforced concrete (RC) constructed facilities; (2) to integrate a drone with tablets/smartphones of both iOS/iPadOS and Android systems for the M1 mode; (3) to integrate an AR-HMD and its gesture control with a laptop for the M2 mode; and (4) to design and develop user-friendly AR user interfaces for both modes. Nine Highly Qualified Personnel (HQP) will be trained in this program, including 3 PhD students, 4 MASc students, and 2 undergraduate research assistants (URAs). Nowadays, the AR applications in the Architecture, Engineering and Construction (AEC) industry mostly rely on pre-loaded data on the AR equipment for object or pattern recognition. This program will bring AR applications to the next level by processing newly captured data in a near-real-time manner, which leads to the importance of the fast defect recognition algorithms to be developed. At the same time, the proposed automated drone inspection mode and the in-person inspection mode with interactive AR-HMD may both revolutionize the current inspection practices and significantly benefit the public and private engineering sectors in Canada and beyond.