AR-integrated intelligent visual inspection system for health monitoring of constructed facilities

建筑、桥梁、管道等已建成设施的老化是一个世界性的关键问题，因为它影响到设施的安全。在加拿大，许多已建成的设施，特别是基础设施位于人口稀少或土著地区，开发自动化智能视觉检测系统可以大大减少人力需求，提高检测效率和有效性。在这方面，本研究项目的目标是开发近实时（NRT）快速缺陷识别算法，并利用增强现实（AR）开发一种集成AR的智能视觉检测系统，用于对已建设施进行近实时健康监测。该智能系统将包括两种模式：(1)自动无人机检查模式（M1）；(2)亲临检验模式（M2）。对于M1， gps制导无人机将沿着预先规划的路线飞行，自动捕获设施表面检查数据（例如，视频或照片），并实时显示在平板电脑或智能手机上，随后将识别到的缺陷立即叠加在显示视图上（即AR效果）。每当一个缺陷被识别，相应的GPS位置将被记录。在无人机完成整个计划路线后，它将自动飞回每个记录的位置，以捕获斑点缺陷周围更详细的数据。对于M2， AR-HMD（头戴式显示器）将捕获检测数据并将其传输到笔记本电脑以快速识别缺陷，然后将识别出的缺陷叠加在AR-HMD上。AR-HMD的交互式手势控制将被纳入开发。为实现这一目标，本项目设定了四个研究目标：(1)开发针对钢筋混凝土（RC）建筑设施的近实时快速缺陷识别算法；(2)将无人机与iOS/iPadOS和Android系统的平板电脑/智能手机集成为M1模式；(3)将AR-HMD及其手势控制与笔记本电脑集成，实现M2模式；(4)为两种模式设计和开发用户友好的AR用户界面。本项目将培养9名高素质人才（HQP），包括3名博士生、4名硕士生和2名本科生研究助理（ura）。目前，建筑、工程和建筑（AEC）行业的AR应用主要依赖于AR设备上的预加载数据进行对象或模式识别。该项目将通过以近乎实时的方式处理新捕获的数据，将增强现实应用提升到一个新的水平，这将导致开发快速缺陷识别算法的重要性。与此同时，拟议的自动无人机检查模式和带有交互式AR-HMD的现场检查模式可能会彻底改变当前的检查实践，并大大有利于加拿大及其他地区的公共和私人工程部门。

The aging of constructed facilities, such as buildings, bridges, pipelines, etc., is a critical issue worldwide, as it affects the safety of the facilities. In Canada, many constructed facilities, especially infrastructure facilities, are in sparsely populated or indigenous areas, and the development of an automated intelligent visual inspection system that can greatly reduce manpower requirement and enhance inspection efficiency and effectiveness is necessary. In this regard, the goal of this research program is to develop near-real-time (NRT) fast defect recognition algorithms and utilize augmented reality (AR) to develop an AR-integrated intelligent visual inspection system for near-real-time health monitoring of constructed facilities. This intelligent system will comprise two modes: (1) automated drone inspection mode (M1); and (2) in-person inspection mode (M2). For M1, a GPS-guided drone will fly along a pre-planned route to automatically capture facility surface inspection data (e.g., videos or photos), and display them on a tablet or smartphone in real time, followed by immediate superimposing of recognized defects on the displayed view (i.e., AR effects). Whenever a defect is identified, the corresponding GPS position will be recorded. After the drone finishes the entire planned route, it will automatically fly back to each of the recorded positions to capture more detailed data around the spotted defect. For M2, an AR-HMD (head-mounted display) will capture inspection data and transmit them to a laptop for fast defect recognition, followed by superimposing recognized defects on AR-HMD. Interactive gesture control of AR-HMD will be incorporated in the development. To achieve the goal of this program, four research objectives are set: (1) to develop near-real-time fast defect recognition algorithms for steel and reinforced concrete (RC) constructed facilities; (2) to integrate a drone with tablets/smartphones of both iOS/iPadOS and Android systems for the M1 mode; (3) to integrate an AR-HMD and its gesture control with a laptop for the M2 mode; and (4) to design and develop user-friendly AR user interfaces for both modes. Nine Highly Qualified Personnel (HQP) will be trained in this program, including 3 PhD students, 4 MASc students, and 2 undergraduate research assistants (URAs). Nowadays, the AR applications in the Architecture, Engineering and Construction (AEC) industry mostly rely on pre-loaded data on the AR equipment for object or pattern recognition. This program will bring AR applications to the next level by processing newly captured data in a near-real-time manner, which leads to the importance of the fast defect recognition algorithms to be developed. At the same time, the proposed automated drone inspection mode and the in-person inspection mode with interactive AR-HMD may both revolutionize the current inspection practices and significantly benefit the public and private engineering sectors in Canada and beyond.