CRII: RI: Modeling and Understanding the Invisible World in Thermal Modality

目前大多数场景建模和理解算法都是针对光照充足的可见光场景和暴露的物体。然而，很少有光线或物体被隐藏的情况是常见的，例如在室外或室内环境中的夜间停电、植物受水压力以及桥面空洞。热像采集长波红外信号生成图像，在交通、农业、监控等领域有着广泛的应用。该项目开发了基于热像的三维建模和理解方法。开发的3D建模技术可以使自动驾驶车辆从热像中检测场景深度，以避免交通事故。有了三维热模型，建筑物中隐藏的燃气管道和桥梁上的裂缝也可以更可靠地被检测出来。具体来说，本项目从以下几个方面解决了三维热传感和理解问题：(1)热图像特征检测和提取，(2)热图像和可见光图像之间的转换，(3)基于图像序列和单幅图像的无监督三维热重建方法，(4)三维热模型理解，特别是对于夜间和不可见物体。该项目中开发的算法将使不可见场景的3D重建和理解任务成为可能。该项目的成果将使白天和白天为可见物体提供的相关服务也可用于看不见的物体和夜间，例如夜间的隐藏物体检查和导航。隐形场景处理能力的增强将提高从白天到夜间以及从裸露物体到隐形物体属性的不同场景的安全和网络安全。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Most current scene modeling and understanding algorithms target at visible scenes with sufficient light supplied and the exposed objects. However, it is common for situations to occur in which there is little light or objects are hidden, such as nighttime in outdoor or indoor environments when the power outage happens, water press of plants, and hollows in the bridge deck. Thermal images capture the long-wave infrared signals to generate images and have broad applications in transportation, agriculture, and surveillance. This project develops 3D modeling and understanding methods based on thermal images. The developed 3D modeling technology can enable autonomous driving vehicles to detect the scene depth from thermal images to avoid traffic accidents. With a 3D thermal model, hidden gas pipes in buildings and cracks in bridges can also be inspected more reliably.Specifically, this project addresses the 3D thermal sensing and understanding issues through the following thrusts: (1) thermal image feature detection and extraction, (2) translation between thermal and visible images, (3) unsupervised 3D thermal reconstruction methods based on both image sequence and a single image, and (4) 3D thermal model understanding, particularly for nighttime and invisible objects. The algorithms developed in this project will enable 3D reconstruction and understanding tasks for invisible scenes. The output from this project will allow related services provided for visible objects and during the daytime to be also available for invisible objects and during nighttime, such as hidden object inspection and navigation in the nighttime. The enhancement of invisible scene processing capability will improve safety and cybersecurity across different scenarios, from daytime to nighttime and from exposed objects to invisible object properties.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.