CAREER: Seeing Through Atmospheric Turbulence: Image Restoration and Understanding using Deep Convolutional Neural Networks

职业：透视大气湍流：使用深度卷积神经网络进行图像恢复和理解

• 批准号: 2045489
• 负责人: Vishal Patel
• 金额: $ 50万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-03-15 至 2026-02-28
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2045489&HistoricalAwards=false
• 关键词: CAREER; Seeing; Through; Atmospheric; Turbulence

Atmospheric turbulence can significantly degrade the quality of images acquired by long-range imaging systems by causing spatially and temporally random fluctuations in the index of refraction of the atmosphere. Variations in the refractive index causes the captured images to be geometrically distorted and blurry. These distortions adversely affect the performance of subsequent computer vision algorithms such as object detection and recognition. Hence, it is important to compensate for the visual degradation in images caused by atmospheric turbulence. Adaptive optics-based techniques can be used to compensate for turbulence effects in images. However, they require large, complex and expensive hardware. On the other hand, image processing-based approaches are cheap and effective. The idea of the proposed approach is to pose the turbulence degraded image restoration problem as a nonlinear regression problem, where the optimal parameters are learned from synthetically generated data. As a function approximator, we propose to use deep convolutional neural networks. The goal of this CAREER project is to develop data-driven learning-based approaches for restoration and understanding of images degraded by atmospheric turbulence. This project will help create new undergraduate/graduate courses on Deep Learning and Image Restoration. Further, this project will impact many diversity outreach activities, including specific outreach to ensure broad participation of women, minorities and disadvantaged groups.Our research will provide a comprehensive framework for restoring and understanding images/videos degraded by atmospheric turbulence. We will significantly innovate in the areas of supervised, semi-supervised and unsupervised image restoration techniques by developing novel end-to-end trainable deep convolutional neural networks and corresponding loss functions. Furthermore, domain transfer learning-based methods for adapting computer vision algorithms such as object detection and segmentation to turbulence-degraded images will be developed. Algorithms that will be developed in this project will significantly enhance the quality of images and videos collected by long-range visible and infrared imagining systems, and will result in improved understanding of images degraded by atmospheric turbulence. The proposed methods will be useful in many applications of remote sensing, long-range surveillance, optical communications, astronomy, road traffic monitoring, underwater imaging, and autonomous navigation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

大气湍流会造成大气折射率在空间和时间上的随机波动，从而大大降低远程成像系统所获得的图像的质量。折射率的变化导致捕获的图像在几何上失真和模糊。这些失真会对后续计算机视觉算法（如对象检测和识别）的性能产生不利影响。因此，重要的是补偿大气湍流引起的图像中的视觉退化。基于自适应光学的技术可用于补偿图像中的湍流效应。然而，它们需要大型、复杂和昂贵的硬件。另一方面，基于图像处理的方法是廉价和有效的。所提出的方法的思想是把湍流退化图像恢复问题作为一个非线性回归问题，其中最佳参数是从综合生成的数据中学习的。作为函数逼近器，我们建议使用深度卷积神经网络。这个CAREER项目的目标是开发基于数据驱动的学习方法，用于恢复和理解大气湍流退化的图像。该项目将帮助创建关于深度学习和图像恢复的新本科生/研究生课程。此外，该项目将影响许多多样性外展活动，包括确保妇女，少数民族和弱势群体广泛参与的具体外展活动。我们的研究将为恢复和理解因大气湍流而退化的图像/视频提供全面的框架。我们将通过开发新型端到端可训练深度卷积神经网络和相应的损失函数，在监督、半监督和无监督图像恢复技术领域进行重大创新。此外，还将开发基于域迁移学习的方法，用于使计算机视觉算法（如物体检测和分割）适应清晰度退化的图像。该项目将开发的算法将大大提高远程可见光和红外成像系统收集的图像和视频的质量，并将导致更好地理解大气湍流退化的图像。所提出的方法将在遥感，远程监视，光通信，天文学，道路交通监测，水下成像和自主navigation.This奖项反映了NSF的法定使命的许多应用是有用的，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

项目成果

Thermal to Visible Image Synthesis Under Atmospheric Turbulence

• DOI: 10.1109/icip46576.2022.9897975
• 发表时间: 2022-04
• 期刊: 2022 IEEE International Conference on Image Processing (ICIP)
• 作者: Kangfu Mei;Yiqun Mei;Vishal M. Patel

SPIN Road Mapper: Extracting Roads from Aerial Images via Spatial and Interaction Space Graph Reasoning for Autonomous Driving

• DOI: 10.1109/icra46639.2022.9812134
• 发表时间: 2021-09
• 期刊: 2022 International Conference on Robotics and Automation (ICRA)
• 作者: W. G. C. Bandara;Jeya Maria Jose Valanarasu;Vishal M. Patel

NBD-GAP: Non-Blind Image Deblurring without Clean Target Images

• DOI: 10.1109/icip46576.2022.9897543
• 发表时间: 2022-09
• 期刊: 2022 IEEE International Conference on Image Processing (ICIP)
• 作者: Nithin Gopalakrishnan Nair;R. Yasarla;Vishal M. Patel

Hyperspectral Pansharpening Based on Improved Deep Image Prior and Residual Reconstruction

• DOI: 10.1109/tgrs.2021.3139292
• 发表时间: 2022-01-01
• 期刊: IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING
• 影响因子: 8.2
• 作者: Bandara, Wele Gedara Chaminda;Valanarasu, Jeya Maria Jose;Patel, Vishal M.
• 通讯作者: Patel, Vishal M.

Escaping Data Scarcity for High-Resolution Heterogeneous Face Hallucination

摆脱高分辨率异构面部幻觉的数据稀缺

• DOI: 10.1109/cvpr52688.2022.01812
• 发表时间: 2022
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR
• 作者: Mei, Yiqun;Guo, Pengfei;Patel, Vishal M.
• 通讯作者: Patel, Vishal M.