CAREER: Perceptual Cameras: Forming Images Through Scene Interpretation

职业:感知相机:通过场景解释形成图像

基本信息

  • 批准号:
    2047359
  • 负责人:
  • 金额:
    $ 50万
  • 依托单位:
  • 依托单位国家:
    美国
  • 项目类别:
    Continuing Grant
  • 财政年份:
    2021
  • 资助国家:
    美国
  • 起止时间:
    2021-02-01 至 2026-01-31
  • 项目状态:
    未结题

项目摘要

Today’s cameras provide a digital window into the real world with broad applications across societal and scientific areas. Despite their remarkably diverse applications, existing cameras are engineered as general-purpose sensing and signal-processing pipelines. This project breaks with this conventional approach and proposes methods to “computationally evolve” the cameras of tomorrow. As such, the results will drastically expand our understanding of how to develop and optimize entire cameras and signal processing chain for a specific application domain, including medical imaging, robotics, scientific imaging, virtual/artificial reality, and self-driving vehicles. We will develop a completely new breed of cameras for these diverse application domains, for example, ones that may consider the scene as part of the camera. The research efforts are tightly integrated with an outreach program that introduces underrepresented and at-risk students in the New Jersey and New York area to science and technology through domain-specific cameras for self-driving vehicles.The research of this project will develop a novel comprehensive learning framework that allows the researchers to reason over a distribution of cameras in a continuous and differentiable fashion. This framework will hinge on a theory that models the illumination, acquisition, processing, scene light transport, and illumination stages of a broad space of cameras. As such, the research team will be able to optimize over the architecture and parameters of full sensing and processing stacks, resulting in fundamentally novel cameras tailored to specific imaging and perception tasks. These new cameras learn to shift complexity between optics, compute, illumination, sensing, and the scene light transport, exploiting the scene and scene semantics as part of the imaging process. This enables unprecedented capabilities for domain-specific imaging in scattering media, ultra-miniaturized learned cameras, neural optical compute, and imaging at ultra-large scales in adverse conditions and ultra-small scales, all of which will be explored in this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
今天的相机提供了一个进入现实世界的数字窗口,在社会和科学领域有着广泛的应用。尽管它们的应用非常多样化,但现有的相机被设计成通用的传感和信号处理管道。这个项目打破了这种传统的方法,提出了“计算进化”未来相机的方法。因此,研究结果将极大地扩展我们对如何为特定应用领域开发和优化整个摄像头和信号处理链的理解,包括医学成像、机器人、科学成像、虚拟/人工现实和自动驾驶汽车。我们将为这些不同的应用领域开发一种全新的相机,例如,可以将场景视为相机的一部分。这项研究工作与一项外展计划紧密结合,该计划通过自动驾驶汽车的特定领域摄像头,向新泽西和纽约地区代表性不足、处境危险的学生介绍科学技术。该项目的研究将开发一种新的综合学习框架,使研究人员能够以连续和可微分的方式对摄像机的分布进行推理。这个框架将依赖于一个理论,该理论模拟了一个广阔空间的相机的照明、采集、处理、场景光传输和照明阶段。因此,研究团队将能够优化全传感和处理堆栈的架构和参数,从而产生针对特定成像和感知任务量身定制的全新相机。这些新相机学习在光学,计算,照明,传感和场景光传输之间转移复杂性,利用场景和场景语义作为成像过程的一部分。这使得散射介质中的特定领域成像、超小型化学习相机、神经光学计算以及在不利条件和超小尺度下的超大尺度成像具有前所未有的能力,所有这些都将在本项目中进行探索。该奖项反映了美国国家科学基金会的法定使命,并通过使用基金会的知识价值和更广泛的影响审查标准进行评估,被认为值得支持。

项目成果

期刊论文数量(5)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
ZeroScatter: Domain Transfer for Long Distance Imaging and Vision through Scattering Media
Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging
Learned Hardware-in-the-loop Phase Retrieval for Holographic Near-Eye Displays
  • DOI:
    10.1145/3414685.3417846
  • 发表时间:
    2020-12-01
  • 期刊:
  • 影响因子:
    6.2
  • 作者:
    Chakravarthula, Praneeth;Tseng, Ethan;Heide, Felix
  • 通讯作者:
    Heide, Felix
Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design
  • DOI:
    10.1145/3446791
  • 发表时间:
    2021-06-01
  • 期刊:
  • 影响因子:
    6.2
  • 作者:
    Tseng, Ethan;Mosleh, Ali;Heide, Felix
  • 通讯作者:
    Heide, Felix
{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

数据更新时间:{{ journalArticles.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ monograph.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ sciAawards.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ conferencePapers.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ patent.updateTime }}

Felix Heide其他文献

Deep-inverse correlography: towards real-time high-resolution non-line-of-sight imaging: erratum
深度逆相关图:实现实时高分辨率非视距成像:勘误表
  • DOI:
    10.1364/optica.391291
  • 发表时间:
    2020
  • 期刊:
  • 影响因子:
    10.4
  • 作者:
    Christopher A. Metzler;Felix Heide;Prasanna V. Rangarajan;M. M. Balaji;A. Viswanath;A. Veeraraghavan;Richard Baraniuk
  • 通讯作者:
    Richard Baraniuk
Gated2Depth: Real-Time Dense Lidar From Gated Images
Gated2Depth:来自门控图像的实时密集激光雷达
Time-of-Flight Imaging
飞行时间成像
Snapshot Difference Imaging using Time-of-Flight Sensors
使用飞行时间传感器进行快照差异成像
  • DOI:
  • 发表时间:
    2017
  • 期刊:
  • 影响因子:
    0
  • 作者:
    C. Callenberg;Felix Heide;Gordon Wetzstein;M. Hullin
  • 通讯作者:
    M. Hullin
Computational methods for aberration correction in simple lens imaging
简单透镜成像像差校正的计算方法
  • DOI:
  • 发表时间:
    2014
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Felix Heide;Mushfiqur Rouf;M. Hullin;B. Labitzke;A. Kolb;W. Heidrich
  • 通讯作者:
    W. Heidrich

Felix Heide的其他文献

{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

{{ truncateString('Felix Heide', 18)}}的其他基金

Collaborative Research: OP: Meta-optical Computational Image Sensors
合作研究:OP:元光学计算图像传感器
  • 批准号:
    2127331
  • 财政年份:
    2021
  • 资助金额:
    $ 50万
  • 项目类别:
    Standard Grant

相似海外基金

Postdoctoral Fellowship: SPRF: Mechanisms Underlying Perceptual Learning of Accented Speech
博士后奖学金:SPRF:口音感知学习的机制
  • 批准号:
    2303087
  • 财政年份:
    2024
  • 资助金额:
    $ 50万
  • 项目类别:
    Fellowship Award
The perceptual mechanisms of optical-flow speed in human and nonhuman primates
人类和非人类灵长类动物光流速度的感知机制
  • 批准号:
    24K16879
  • 财政年份:
    2024
  • 资助金额:
    $ 50万
  • 项目类别:
    Grant-in-Aid for Early-Career Scientists
Human Scheduling of Perceptual Tasks
人类感知任务的调度
  • 批准号:
    DP240100979
  • 财政年份:
    2024
  • 资助金额:
    $ 50万
  • 项目类别:
    Discovery Projects
NSF-BSF: Mechanisms of Perceptual Enhancement by Action Preparation
NSF-BSF:通过行动准备增强知觉的机制
  • 批准号:
    2341363
  • 财政年份:
    2024
  • 资助金额:
    $ 50万
  • 项目类别:
    Standard Grant
CPS: Medium: Collaborative Research: Robust Sensing and Learning for Autonomous Driving Against Perceptual Illusion
CPS:中:协作研究:针对自动驾驶对抗知觉错觉的鲁棒感知和学习
  • 批准号:
    2235231
  • 财政年份:
    2023
  • 资助金额:
    $ 50万
  • 项目类别:
    Standard Grant
Learning mechanisms for perceptual decisions in biological and artificial neural systems
生物和人工神经系统中感知决策的学习机制
  • 批准号:
    BB/X013235/1
  • 财政年份:
    2023
  • 资助金额:
    $ 50万
  • 项目类别:
    Research Grant
Non-sensory Circuits for Auditory Perceptual Learning
用于听觉感知学习的非感觉回路
  • 批准号:
    10563542
  • 财政年份:
    2023
  • 资助金额:
    $ 50万
  • 项目类别:
Rotation 1: Biological plausible models of visual perceptual learning
旋转 1:视觉感知学习的生物学合理模型
  • 批准号:
    2887737
  • 财政年份:
    2023
  • 资助金额:
    $ 50万
  • 项目类别:
    Studentship
FMSG: Cyber: Perceptual and Cognitive Additive Manufacturing (PCAM)
FMSG:网络:感知和认知增材制造 (PCAM)
  • 批准号:
    2229155
  • 财政年份:
    2023
  • 资助金额:
    $ 50万
  • 项目类别:
    Standard Grant
From lab to math classroom: Utilizing eye gaze and cognitive control tasks to examine the effects of perceptual cues and structure on mathematical performance
从实验室到数学课堂:利用目光注视和认知控制任务来检查感知线索和结构对数学表现的影响
  • 批准号:
    2320053
  • 财政年份:
    2023
  • 资助金额:
    $ 50万
  • 项目类别:
    Standard Grant
{{ showInfoDetail.title }}

作者:{{ showInfoDetail.author }}

知道了