A semi-supervised self-training method to develop assistive intelligence for segmenting multiclass bridge elements from inspection videos
A semi-supervised self-training method to develop assistive intelligence for segmenting multiclass bridge elements from inspection videos
复制标题
DOI：
10.1177/14759217211010422
复制
发表时间：
2021-05
期刊：
影响因子：
--
通讯作者：
Muhammad Monjurul Karim;Ruwen Qin;Genda Chen;Zhaozheng Yin
中科院分区：
文献类型：
--
作者：
Muhammad Monjurul Karim;Ruwen Qin;Genda Chen;Zhaozheng Yin
桥梁检查是保护和修复交通基础设施以延长其使用寿命的重要一步。移动的机器人技术的进步允许快速收集大量的检测视频数据。然而，数据主要是复杂场景的图像，其中各种结构元素的桥与杂乱的背景混合。协助桥梁检查员从庞大复杂的视频数据中提取桥梁的结构元素，并按类别进行分类，为检查员进行元素检查以确定桥梁的状况做好准备。本文的目的是开发一种辅助智能模型，用于从空中检测平台捕获的检测视频中分割多类桥梁构件。通过检查员标记的小的初始训练数据集，在大型公共数据集上预训练的基于掩码区域的卷积神经网络被转移到多类桥梁元素分割的新任务中。此外，时间相干性分析试图恢复假阴性并识别神经网络可以学习改进的弱点。此外，开发了一种半监督的自我训练方法，以使有经验的检查员迭代地改进网络。评估开发的深度神经网络的定量和定性结果表明，所提出的方法可以利用少量的时间和经验丰富的检查员的指导（标记66张图像的3.58小时）来构建性能优异的网络（91.8%的准确率，93.6%的召回率和92.7%的f1分数）。重要的是，本文说明了一种方法，利用桥梁专业人员的领域知识和经验到计算智能模型，有效地适应不同的桥梁模型在国家桥梁库存。
Bridge inspection is an important step in preserving and rehabilitating transportation infrastructure for extending their service lives. The advancement of mobile robotic technology allows the rapid collection of a large amount of inspection video data. However, the data are mainly the images of complex scenes, wherein a bridge of various structural elements mix with a cluttered background. Assisting bridge inspectors in extracting structural elements of bridges from the big complex video data, and sorting them out by classes, will prepare inspectors for the element-wise inspection to determine the condition of bridges. This article is motivated to develop an assistive intelligence model for segmenting multiclass bridge elements from the inspection videos captured by an aerial inspection platform. With a small initial training dataset labeled by inspectors, a Mask Region-based Convolutional Neural Network pre-trained on a large public dataset was transferred to the new task of multiclass bridge element segmentation. Besides, the temporal coherence analysis attempts to recover false negatives and identify the weakness that the neural network can learn to improve. Furthermore, a semi-supervised self-training method was developed to engage experienced inspectors in refining the network iteratively. Quantitative and qualitative results from evaluating the developed deep neural network demonstrate that the proposed method can utilize a small amount of time and guidance from experienced inspectors (3.58 h for labeling 66 images) to build the network of excellent performance (91.8% precision, 93.6% recall, and 92.7% f1-score). Importantly, the article illustrates an approach to leveraging the domain knowledge and experiences of bridge professionals into computational intelligence models to efficiently adapt the models to varied bridges in the National Bridge Inventory.