Collaborative Research: CIF: Medium: Snapshot Computational Imaging with Metaoptics

光与世界上的物质沿着它的许多轴相互作用，包括空间、时间、角度、光谱和偏振。在细粒度尺度上测量这些相互作用是许多科学工作中的关键技术，包括生命科学，遥感，安全和取证以及增强和虚拟现实。然而，传统的图像传感器仅捕获二维空间变化。因此，光的其他属性通过将它们嵌入空间维度内来测量（例如，空间上散布在传感器上的拜耳颜色或偏振滤波器马赛克）或顺序地捕获它们（例如，获取时间维度上的连续视频帧或频谱维度上的连续高光谱分量）。然而，使用空间平铺的快照方法虽然简单，但会导致混叠伪影，并且总是需要昂贵的制造技术（将颜色/偏振滤波器粘合到传感器阵列）。另一方面，顺序测量必然会产生运动伪影和较低的帧速率。相比之下，该项目开发了快照计算相机，用于通过利用超光学的最新进展来捕获沿着光的各个维度的信息，即，光学器件使用亚波长纳米结构来操纵光的特性，例如相位、波长、振幅或偏振，其控制程度在传统折射光学中是不可行的。该项目的重点是开发基于超光学的成像系统，采用频域多路复用，而不是空间拼接或顺序成像。这种频率复用技术需要对现有成像系统进行最小的改变，同时能够以最小的混叠伪影进行多个维度的快照测量。该项目将侧重于实现快照计算成像系统的三个主要目标。第一个目标是基于计算相机的渲染算法以及此类相机的组合和可区分版本构建模拟器，以有效地模拟光的各种维度，包括飞行时间，光谱和偏振。同时，将建立一个可扩展的、可微分的超光学模拟器，当光与小于光的波长的超光学纳米结构相互作用时，该模拟器可以处理波效应。第二个目标是通过利用在第一个目标中开发的模拟器来设计频率复用快照相机，从而导致基于超光学的相机能够以高数值孔径捕获大景深的强度，从而实现具有低操作功率的紧凑成像系统。第三个目标是通过设计和构建实验室原型并将其与当前最先进的成像仪进行比较来展示快照相机的优势。该项目的成果将影响到各个学科，包括计算机图形学、光学、计算成像和生物医学成像。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Light interacts richly with materials in the world along its many axes, including spatial, temporal, angular, spectral, and polarization. Measuring these interactions at a fine-grained scale is a key enabling technology in numerous scientific endeavors, including life sciences, remote sensing, security and forensics, and augmented and virtual reality. Yet, traditional image sensors capture only two-dimensional spatial variations. Therefore, other properties of light are measured by either embedding them within the spatial dimensions (e.g., Bayer-color or polarization-filter mosaics spatially spread over the sensor) or capturing them sequentially (e.g., acquiring consecutive video frames in the time dimension or consecutive hyperspectral components in the spectral dimension). However, snapshot approaches that use spatial tiling, while simple, lead to aliasing artifacts and invariably require expensive manufacturing techniques (bonding color/polarization filters to the sensor array). On the other hand, sequential measurements entail motion artifacts and lower frame rates. In contrast, this project develops snapshot computational cameras for capturing information along light's various dimensions by leveraging recent advances in metaoptics, i.e., optical devices that use sub-wavelength nano-structures to manipulate light characteristics - such as phase, wavelength, amplitude, or polarization - with a degree of control not feasible in traditional refractive optics. The project focuses on developing metaoptics-based imaging systems with frequency-domain multiplexing instead of spatial tiling or sequential imaging. Such frequency-multiplexed techniques require minimal changes to existing imaging systems while enabling snapshot measurements of multiple dimensions with minimal aliasing artifacts. The project will focus on three main objectives to achieve snapshot computational-imaging systems. The first objective is to build simulators based on rendering algorithms for computational cameras, as well as combinations and differentiable versions of such cameras, to efficiently simulate the various dimensions of light, including time-of-flight, spectrum, and polarization. In tandem, a scalable, differentiable metaoptics simulator will be built that can handle wave effects as light interacts with the metaoptical nano-structures which are smaller than the wavelength of the light. The second objective is to design frequency-multiplexed snapshot cameras by leveraging the simulators developed in the first objective, leading to metaoptics-based cameras that enable capturing intensity over large depths of field with high numerical aperture, thereby achieving compact imaging systems with low operational power. The third objective is to demonstrate the advantages of snapshot cameras by designing and building lab prototypes and comparing them against current state-of-the-art imagers. The outcomes of this project will impact various disciplines, including computer graphics, optics, computational imaging, and biomedical imaging.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.