CAREER: Perceptual Cameras: Forming Images Through Scene Interpretation

今天的相机提供了一个了解现实世界的数字窗口，在社会和科学领域有着广泛的应用。尽管它们的应用非常多样化，但现有的摄像头被设计成通用的传感和信号处理管道。这个项目打破了这种传统的方法，提出了“计算进化”明天的相机的方法。因此，研究结果将极大地扩大我们对如何为特定应用领域开发和优化整个相机和信号处理链的理解，这些领域包括医学成像、机器人、科学成像、虚拟/人工现实和自动驾驶车辆。我们将为这些不同的应用领域开发一种全新的相机类型，例如，可能会将场景视为相机的一部分。研究工作与一个外展项目紧密结合在一起，该项目通过针对自动驾驶汽车的特定领域的摄像头，向新泽西州和纽约地区未被充分代表的和有风险的学生介绍科学和技术。该项目的研究将开发一个新颖的综合学习框架，使研究人员能够以连续和可区分的方式对摄像头的分布进行推理。这一框架将建立在一个理论的基础上，该理论对广阔的摄像机空间的照明、采集、处理、场景光传输和照明阶段进行建模。因此，研究团队将能够对完整的传感和处理堆栈的架构和参数进行优化，从而从根本上为特定的成像和感知任务量身定做新型相机。这些新相机学会了在光学、计算、照明、传感和场景光传输之间转移复杂性，将场景和场景语义作为成像过程的一部分。这使得在散射介质、超微型学习相机、神经光学计算以及在恶劣条件和超小规模下的超大规模成像方面具有前所未有的能力，所有这些都将在本项目中进行探索。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Today’s cameras provide a digital window into the real world with broad applications across societal and scientific areas. Despite their remarkably diverse applications, existing cameras are engineered as general-purpose sensing and signal-processing pipelines. This project breaks with this conventional approach and proposes methods to “computationally evolve” the cameras of tomorrow. As such, the results will drastically expand our understanding of how to develop and optimize entire cameras and signal processing chain for a specific application domain, including medical imaging, robotics, scientific imaging, virtual/artificial reality, and self-driving vehicles. We will develop a completely new breed of cameras for these diverse application domains, for example, ones that may consider the scene as part of the camera. The research efforts are tightly integrated with an outreach program that introduces underrepresented and at-risk students in the New Jersey and New York area to science and technology through domain-specific cameras for self-driving vehicles.The research of this project will develop a novel comprehensive learning framework that allows the researchers to reason over a distribution of cameras in a continuous and differentiable fashion. This framework will hinge on a theory that models the illumination, acquisition, processing, scene light transport, and illumination stages of a broad space of cameras. As such, the research team will be able to optimize over the architecture and parameters of full sensing and processing stacks, resulting in fundamentally novel cameras tailored to specific imaging and perception tasks. These new cameras learn to shift complexity between optics, compute, illumination, sensing, and the scene light transport, exploiting the scene and scene semantics as part of the imaging process. This enables unprecedented capabilities for domain-specific imaging in scattering media, ultra-miniaturized learned cameras, neural optical compute, and imaging at ultra-large scales in adverse conditions and ultra-small scales, all of which will be explored in this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.