RI: Small: Learning to See Through Atmospheric Turbulence

RI：小：学习看穿大气湍流

• 批准号: 2133032
• 负责人: Stanley Chan
• 金额: $ 50万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2025-12-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2133032&HistoricalAwards=false
• 关键词: RI; Small; Learning; See; Through

For a variety of long-range imaging systems in autonomous vehicles, surveillance, and defense, restoring images that are distorted by atmospheric turbulence is inevitable. However, unlike the better-known image restoration problems such as denoising and deblurring, recovering turbulence distorted images is considerably more difficult because of the physics involved. On one hand, the image formation process due to a turbulent medium is described by a sequence of wave equations of diffraction and phase distortion. The lack of a simple forward model makes the inverse problem difficult to formulate and solve. On the other hand, while deep learning algorithms have produced promising results in many disciplines, the disparity between these generic models and the specific turbulence physics makes the resulting methods lack generalizability, explainability, and robustness. The goal of this proposal is to bridge the gap between turbulence physics and deep learning algorithms. The approach is to ground the algorithmic designs on physics by developing new forward models, reconstruction algorithms, training schemes that improve consistency, and benchmark evaluation. By improving the image restoration capability, the project will enable a wide range of imaging applications and software products that, in turn, improve object detection, biometric analysis, and navigation. For mission-critical applications such as defense, the integration of physics and algorithms will provide more consistent and trustworthy information for decision-making. The project also trains next-generation imaging scientists that will provide the necessary workforce to the United States.To accomplish the goal of the project, four objectives will be pursued. (1) To develop a new forward model that has low complexity, adheres to physics, and is differentiable in the sense of backpropagation. The new model will fill the critical need for a viable turbulence simulator that can generate data at a large scale for training and testing. (2) To develop a new image restoration algorithm by integrating the forward model, lucky imaging, and end-to-end neural networks. Specifically, a new strategy for feature matching in the presence of turbulence and noise will be developed, and inverse optimization will be formulated via the concept of unrolled neural networks. It is anticipated that the new techniques will enable the imaging of small and moving objects. (3) To develop a new training scheme that improves the consistency of the algorithm from one turbulence condition to another, by optimally allocating the training samples according to the turbulence strengths. (4) To establish a benchmark evaluation system by building controllable experimental setups and collecting real data. On the education front, the project aims to promote the exchange of knowledge across physics and deep learning by developing tutorials in major computer vision and optics conferences; disseminating educational materials to the general public through classes and books; delivering codes and datasets to support reproducible research. The project will promote STEM education by offering image processing and machine learning to high school students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

对于自动驾驶汽车、监视和防御中的各种远程成像系统，恢复因大气湍流而失真的图像是不可避免的。然而，与众所周知的图像恢复问题，如去噪和去模糊，恢复湍流失真的图像是相当困难的，因为涉及的物理。一方面，由于湍流介质的图像形成过程是由一系列的衍射和相位畸变的波动方程描述。缺乏一个简单的正向模型使得逆问题难以制定和解决。另一方面，虽然深度学习算法在许多学科中产生了有希望的结果，但这些通用模型与特定湍流物理学之间的差异使得所产生的方法缺乏可推广性，可解释性和鲁棒性。该提案的目标是弥合湍流物理学和深度学习算法之间的差距。该方法是通过开发新的前向模型、重建算法、提高一致性的训练方案和基准评估，将算法设计建立在物理基础上。通过提高图像恢复能力，该项目将实现广泛的成像应用程序和软件产品，从而改善物体检测，生物特征分析和导航。对于国防等关键任务应用，物理和算法的集成将为决策提供更一致和更可靠的信息。该项目还培训下一代成像科学家，为美国提供必要的劳动力。(1)开发一种新的前向模型，该模型具有低复杂性，遵循物理学，并且在反向传播意义上是可微的。新模型将满足对可行的湍流模拟器的迫切需求，该模拟器可以大规模生成用于训练和测试的数据。(2)将前向模型、幸运成像和端到端神经网络相结合，提出一种新的图像复原算法。具体而言，将开发一种新的策略，在湍流和噪声的存在下进行特征匹配，并通过展开的神经网络的概念制定逆优化。预计新技术将能够对小型和移动物体进行成像。(3)提出一种新的训练方案，根据湍流强度对训练样本进行最优分配，从而提高算法在不同湍流条件下的一致性。(4)通过建立可控的实验装置和收集真实的数据，建立基准评价体系。在教育方面，该项目旨在通过在主要的计算机视觉和光学会议上开发教程来促进物理学和深度学习方面的知识交流;通过课程和书籍向公众传播教育材料;提供代码和数据集以支持可复制的研究。 该项目将通过向高中生提供图像处理和机器学习来促进STEM教育。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

On the Insensitivity of Bit Density to Read Noise in One-Bit Quanta Image Sensors

• DOI: 10.1109/jsen.2023.3235493
• 发表时间: 2022-03
• 期刊: IEEE Sensors Journal
• 影响因子: 4.3
• 作者: Stanley H. Chan

What Does a One-Bit Quanta Image Sensor Offer?

一位 Quanta 图像传感器提供什么功能？

• DOI: 10.1109/tci.2022.3202012
• 发表时间: 2022
• 期刊: IEEE Transactions on Computational Imaging
• 影响因子: 5.4
• 作者: Chan, Stanley H.

Accelerating Atmospheric Turbulence Simulation via Learned Phase-to-Space Transform

通过学习的相空间变换加速大气湍流模拟

• DOI: 10.1109/iccv48922.2021.01449
• 发表时间: 2021
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision (ICCV
• 作者: Mao, Zhiyuan;Chimitt, Nicholas;Chan, Stanley H.
• 通讯作者: Chan, Stanley H.

Tilt-Then-Blur or Blur-Then-Tilt? Clarifying the Atmospheric Turbulence Model

倾斜然后模糊还是模糊然后倾斜？

• DOI: 10.1109/lsp.2022.3200551
• 发表时间: 2022
• 期刊: IEEE Signal Processing Letters
• 影响因子: 3.9
• 作者: Chan, Stanley H.

Real-Time Dense Field Phase-to-Space Simulation of Imaging Through Atmospheric Turbulence

大气湍流成像的实时密集场相空间模拟

• DOI: 10.1109/tci.2022.3226293
• 发表时间: 2022
• 期刊: IEEE Transactions on Computational Imaging
• 影响因子: 5.4
• 作者: Chimitt, Nicholas;Zhang, Xingguang;Mao, Zhiyuan;Chan, Stanley H.
• 通讯作者: Chan, Stanley H.