RI: Small: Learning to See Through Atmospheric Turbulence

对于自动驾驶车辆、监视和国防中的各种远程成像系统，恢复因大气湍流而扭曲的图像是不可避免的。然而，与众所周知的图像恢复问题不同，如去噪和去模糊，恢复湍流扭曲的图像要困难得多，因为涉及到物理问题。一方面，湍流介质的成像过程可以用一系列绕射和位相失真的波动方程来描述。由于缺乏简单的正演模型，使得反问题的表述和求解变得困难。另一方面，尽管深度学习算法已经在许多学科中产生了令人振奋的结果，但这些通用模型与特定湍流物理之间的差异使得所产生的方法缺乏普适性、可解释性和稳健性。这项提议的目标是弥合湍流物理和深度学习算法之间的差距。方法是通过开发新的正演模型、重建算法、提高一致性的培训方案和基准评估，将算法设计建立在物理学基础上。通过提高图像恢复能力，该项目将支持广泛的成像应用程序和软件产品，这些应用程序和软件产品反过来又改进了对象检测、生物特征分析和导航。对于国防等任务关键型应用，物理和算法的结合将为决策提供更一致、更可信的信息。该项目还培养下一代成像科学家，为美国提供必要的劳动力。为了实现该项目的目标，将实现四个目标。(1)发展一种新的正演模型，该模型具有低复杂度、符合物理学和反向传播意义上的可微性。新模型将满足对可行的湍流模拟器的迫切需求，该模拟器可以生成用于训练和测试的大规模数据。(2)将前向模型、瑞幸成像和端到端神经网络相结合，提出了一种新的图像恢复算法。具体地说，将开发一种在存在湍流和噪声的情况下进行特征匹配的新策略，并将通过展开神经网络的概念来表示逆优化。预计新技术将能够对小的和运动的物体进行成像。(3)开发了一种新的训练方案，通过根据湍流强度对训练样本进行优化分配，提高了算法从一种湍流状态到另一种湍流状态的一致性。(4)通过建立可控的实验装置和收集真实数据，建立基准评估体系。在教育方面，该项目旨在通过编写主要计算机视觉和光学会议的教程；通过课堂和书籍向公众传播教育材料；提供代码和数据集，支持可重复的研究，促进物理和深度学习之间的知识交流。该项目将通过向高中生提供图像处理和机器学习来促进STEM教育。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

For a variety of long-range imaging systems in autonomous vehicles, surveillance, and defense, restoring images that are distorted by atmospheric turbulence is inevitable. However, unlike the better-known image restoration problems such as denoising and deblurring, recovering turbulence distorted images is considerably more difficult because of the physics involved. On one hand, the image formation process due to a turbulent medium is described by a sequence of wave equations of diffraction and phase distortion. The lack of a simple forward model makes the inverse problem difficult to formulate and solve. On the other hand, while deep learning algorithms have produced promising results in many disciplines, the disparity between these generic models and the specific turbulence physics makes the resulting methods lack generalizability, explainability, and robustness. The goal of this proposal is to bridge the gap between turbulence physics and deep learning algorithms. The approach is to ground the algorithmic designs on physics by developing new forward models, reconstruction algorithms, training schemes that improve consistency, and benchmark evaluation. By improving the image restoration capability, the project will enable a wide range of imaging applications and software products that, in turn, improve object detection, biometric analysis, and navigation. For mission-critical applications such as defense, the integration of physics and algorithms will provide more consistent and trustworthy information for decision-making. The project also trains next-generation imaging scientists that will provide the necessary workforce to the United States.To accomplish the goal of the project, four objectives will be pursued. (1) To develop a new forward model that has low complexity, adheres to physics, and is differentiable in the sense of backpropagation. The new model will fill the critical need for a viable turbulence simulator that can generate data at a large scale for training and testing. (2) To develop a new image restoration algorithm by integrating the forward model, lucky imaging, and end-to-end neural networks. Specifically, a new strategy for feature matching in the presence of turbulence and noise will be developed, and inverse optimization will be formulated via the concept of unrolled neural networks. It is anticipated that the new techniques will enable the imaging of small and moving objects. (3) To develop a new training scheme that improves the consistency of the algorithm from one turbulence condition to another, by optimally allocating the training samples according to the turbulence strengths. (4) To establish a benchmark evaluation system by building controllable experimental setups and collecting real data. On the education front, the project aims to promote the exchange of knowledge across physics and deep learning by developing tutorials in major computer vision and optics conferences; disseminating educational materials to the general public through classes and books; delivering codes and datasets to support reproducible research. The project will promote STEM education by offering image processing and machine learning to high school students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.