CCSS: Uncertainty-Aware Computational Imaging in the Wild: a Bayesian Deep Learning Approach in the Latent Space

从显微镜和断层扫描到生物识别和监控，计算成像（CI）在各个科学和工程领域极大地扩展了人类的视觉能力。与需要昂贵光学器件的基于硬件的解决方案不同，由深度学习驱动的数据驱动或计算方法推动了人工智能计算成像系统的发展。新兴的贝叶斯深度学习范式赋予下一代CI系统更大的灵活性，以应对野外的各种不确定性——无论是与图像采集的敌对条件（如移动平台或恶劣天气）有关，还是与人类无意的错误（如哈勃太空望远镜发射升空后主镜出现故障）有关。开发不确定性感知CI系统对科学探索（例如，微观和天文等极端尺度的成像）和我们的日常生活（例如，智能手机应用程序）都有广泛的影响。该项目旨在采用贝叶斯深度学习（BDL）方法来模拟现实世界成像场景中的不确定性。该项目带来的一个重要的新见解是将基于流的不确定性核估计（用于似然建模）与基于记忆的潜在空间不确定性图像生成（用于先验建模）统一起来。在完全贝叶斯框架下，通过重新参数化简化建模过程，极大地促进了不确定性不可避免的现实场景下的退化学习和图像重建。本文的研究包括三个方面：1)基于流的隐空间非均匀核估计；2)基于记忆增强的图像重建深度生成模型；3)深层组织成像的实际应用，如超分辨率显微镜。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

From microscopy and tomography to biometrics and surveillance, computational imaging (CI) has greatly expanded mankind’s vision capability in various scientific and engineering fields. In contrast to hardware-based solutions that require expensive optics, data-driven or computational approaches powered by deep learning have fueled the development of AI-enabled computational imaging systems. The emerging paradigm of Bayesian deep learning endows the next-generation CI systems more flexibility to handle various uncertainties in the wild – no matter whether they are related to the adversarial conditions of image acquisition (e.g., moving platform or bad weather) or unintentional mistakes made by humans (e.g., the mal-functioned main mirror of Hubble Space Telescope after the launch into the space). Developing uncertainty-aware CI systems have a wide range of impact on both scientific exploration (e.g., imaging at extreme scales such as microscopic and astronomical) and our daily lives (e.g., smartphone applications).This project aims at taking a Bayesian deep learning (BDL) approach to modeling uncertainty in real-world imaging scenarios. An important new insight brought about by this project is to unify the flow-based uncertainty kernel estimation (for likelihood modeling) with memory-based uncertainty image generation in latent space (for prior modeling). Under a fully Bayesian framework but using reparameterization to simplify the modeling process, the proposed BDL greatly facilitates both degradation learning and image reconstruction in the realistic scenario when uncertainty is inevitable. The proposed research consists of three tasks: 1) Flow-based Nonuniform Kernel Estimation in the Latent Space; 2) Memory-enhanced Deep Generative Models for Image Reconstruction; and 3) Real-world Applications in Deep Tissue Imaging such as Superresolution Microscopy.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.