CAREER: Perceptual Cameras: Forming Images Through Scene Interpretation

今天的相机提供了一个进入现实世界的数字窗口，在社会和科学领域有着广泛的应用。尽管它们的应用非常多样化，但现有的相机被设计成通用的传感和信号处理管道。这个项目打破了这种传统的方法，提出了“计算进化”未来相机的方法。因此，研究结果将极大地扩展我们对如何为特定应用领域开发和优化整个摄像头和信号处理链的理解，包括医学成像、机器人、科学成像、虚拟/人工现实和自动驾驶汽车。我们将为这些不同的应用领域开发一种全新的相机，例如，可以将场景视为相机的一部分。这项研究工作与一项外展计划紧密结合，该计划通过自动驾驶汽车的特定领域摄像头，向新泽西和纽约地区代表性不足、处境危险的学生介绍科学技术。该项目的研究将开发一种新的综合学习框架，使研究人员能够以连续和可微分的方式对摄像机的分布进行推理。这个框架将依赖于一个理论，该理论模拟了一个广阔空间的相机的照明、采集、处理、场景光传输和照明阶段。因此，研究团队将能够优化全传感和处理堆栈的架构和参数，从而产生针对特定成像和感知任务量身定制的全新相机。这些新相机学习在光学，计算，照明，传感和场景光传输之间转移复杂性，利用场景和场景语义作为成像过程的一部分。这使得散射介质中的特定领域成像、超小型化学习相机、神经光学计算以及在不利条件和超小尺度下的超大尺度成像具有前所未有的能力，所有这些都将在本项目中进行探索。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Today’s cameras provide a digital window into the real world with broad applications across societal and scientific areas. Despite their remarkably diverse applications, existing cameras are engineered as general-purpose sensing and signal-processing pipelines. This project breaks with this conventional approach and proposes methods to “computationally evolve” the cameras of tomorrow. As such, the results will drastically expand our understanding of how to develop and optimize entire cameras and signal processing chain for a specific application domain, including medical imaging, robotics, scientific imaging, virtual/artificial reality, and self-driving vehicles. We will develop a completely new breed of cameras for these diverse application domains, for example, ones that may consider the scene as part of the camera. The research efforts are tightly integrated with an outreach program that introduces underrepresented and at-risk students in the New Jersey and New York area to science and technology through domain-specific cameras for self-driving vehicles.The research of this project will develop a novel comprehensive learning framework that allows the researchers to reason over a distribution of cameras in a continuous and differentiable fashion. This framework will hinge on a theory that models the illumination, acquisition, processing, scene light transport, and illumination stages of a broad space of cameras. As such, the research team will be able to optimize over the architecture and parameters of full sensing and processing stacks, resulting in fundamentally novel cameras tailored to specific imaging and perception tasks. These new cameras learn to shift complexity between optics, compute, illumination, sensing, and the scene light transport, exploiting the scene and scene semantics as part of the imaging process. This enables unprecedented capabilities for domain-specific imaging in scattering media, ultra-miniaturized learned cameras, neural optical compute, and imaging at ultra-large scales in adverse conditions and ultra-small scales, all of which will be explored in this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.