Collaborative Research: CIF: Medium: Snapshot Computational Imaging with Metaoptics

光沿着它的许多轴与世界上的物质丰富地相互作用，包括空间、时间、角度、光谱和偏振。在细粒度尺度上测量这些相互作用是许多科学努力中的关键支持技术，包括生命科学、遥感、安全和取证以及增强和虚拟现实。然而，传统的图像传感器只能捕捉二维空间变化。因此，测量光的其他属性，要么将它们嵌入空间维度（例如，拜耳颜色或偏振滤光片马赛克在传感器上空间分布），要么顺序捕获它们（例如，在时间维度中获取连续视频帧，或在光谱维度中获取连续高光谱分量）。然而，使用空间平铺的快照方法虽然简单，但会导致混叠伪影，并且总是需要昂贵的制造技术（将彩色/偏振滤光片粘合到传感器阵列）。另一方面，顺序测量需要运动伪影和较低的帧率。相比之下，该项目开发了快照计算相机，通过利用元光学的最新进展，捕捉光的各种维度的信息，即使用亚波长纳米结构来操纵光特性（如相位、波长、振幅或偏振）的光学设备，其控制程度在传统折射光学中是不可行的。该项目的重点是开发基于元光学的成像系统，采用频域多路复用，而不是空间平铺或顺序成像。这种频率复用技术需要对现有成像系统进行最小的更改，同时能够以最小的混叠伪影实现多维快照测量。该项目将集中于实现快照计算成像系统的三个主要目标。第一个目标是建立基于计算相机渲染算法的模拟器，以及这些相机的组合和可微分版本，以有效地模拟光的各种维度，包括飞行时间、光谱和偏振。同时，一个可扩展的、可微分的元光学模拟器将被建立，它可以处理光与比光波长小的元光学纳米结构相互作用时的波动效应。第二个目标是通过利用在第一个目标中开发的模拟器来设计多频快照相机，从而导致基于元光学的相机，能够在大景深和高数值孔径上捕获强度，从而实现低操作功耗的紧凑成像系统。第三个目标是通过设计和构建实验室原型来展示快照相机的优势，并将它们与当前最先进的成像仪进行比较。该项目的成果将影响多个学科，包括计算机图形学、光学、计算成像和生物医学成像。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Light interacts richly with materials in the world along its many axes, including spatial, temporal, angular, spectral, and polarization. Measuring these interactions at a fine-grained scale is a key enabling technology in numerous scientific endeavors, including life sciences, remote sensing, security and forensics, and augmented and virtual reality. Yet, traditional image sensors capture only two-dimensional spatial variations. Therefore, other properties of light are measured by either embedding them within the spatial dimensions (e.g., Bayer-color or polarization-filter mosaics spatially spread over the sensor) or capturing them sequentially (e.g., acquiring consecutive video frames in the time dimension or consecutive hyperspectral components in the spectral dimension). However, snapshot approaches that use spatial tiling, while simple, lead to aliasing artifacts and invariably require expensive manufacturing techniques (bonding color/polarization filters to the sensor array). On the other hand, sequential measurements entail motion artifacts and lower frame rates. In contrast, this project develops snapshot computational cameras for capturing information along light's various dimensions by leveraging recent advances in metaoptics, i.e., optical devices that use sub-wavelength nano-structures to manipulate light characteristics - such as phase, wavelength, amplitude, or polarization - with a degree of control not feasible in traditional refractive optics. The project focuses on developing metaoptics-based imaging systems with frequency-domain multiplexing instead of spatial tiling or sequential imaging. Such frequency-multiplexed techniques require minimal changes to existing imaging systems while enabling snapshot measurements of multiple dimensions with minimal aliasing artifacts. The project will focus on three main objectives to achieve snapshot computational-imaging systems. The first objective is to build simulators based on rendering algorithms for computational cameras, as well as combinations and differentiable versions of such cameras, to efficiently simulate the various dimensions of light, including time-of-flight, spectrum, and polarization. In tandem, a scalable, differentiable metaoptics simulator will be built that can handle wave effects as light interacts with the metaoptical nano-structures which are smaller than the wavelength of the light. The second objective is to design frequency-multiplexed snapshot cameras by leveraging the simulators developed in the first objective, leading to metaoptics-based cameras that enable capturing intensity over large depths of field with high numerical aperture, thereby achieving compact imaging systems with low operational power. The third objective is to demonstrate the advantages of snapshot cameras by designing and building lab prototypes and comparing them against current state-of-the-art imagers. The outcomes of this project will impact various disciplines, including computer graphics, optics, computational imaging, and biomedical imaging.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.