CAREER: Cameras and Algorithms that turn Rays Efficiently into Everyday Reconstructions

该奖项部分由2021年美国救援计划法案（公法117-2）资助。该项目旨在使人们能够比目前更准确，更有效，更灵活地捕捉现实世界场景的数字版本，以便它可以成为一个简单的日常任务。在计算机视觉中，捕获真实的世界被称为重建，这是一个核心挑战，需要估计场景中的3D形状，运动，物体材料和照明。成功的重建可以向智能系统提供独立于照明条件的对象的空间和几何信息，使得它们可以使用该信息来从不同的视角推理或创建这些对象的应用，例如，在虚拟和增强现实中。该项目将科学地研究如何通过以符合图像形成物理学的方式组合来自不同类型相机的信号来克服重建的挑战。该项目将通过开设新的跨学科课程和促进不同外联活动的多样性，支持我们的K-12 AI 4ALL本地多样性的努力，并参加在布朗的谢里丹教学和学习中心的包容性教学研讨会。 为了帮助克服被动RGB相机场景重建的不适定问题，该项目有三个重点领域：1）研究通过基于物理的图像形成模型集成多种信号的新相机系统。现有的平台通常处理一种模态和频率（可见光），但该项目旨在将联合收割机结合可见光，飞行时间和事件相机，以平衡每个相机的负面影响，并产生单个相机无法产生的质量信号：高时空分辨率3D视频。2)研究照明和材料分解通过更好的捕获，采样，并通过新的快速视图合成方法，适用于表示入射照明从异构全向相机重建。这将使用从基于物理的反射率模型的因子分解中学习的材料先验，该模型可以利用捕获的全部和部分全向样本。3)研究混合表示，优化和机器学习方法，包括基于深度传感器的可靠稀疏采样的初始化，通过基于物理的自监督变换来约束优化，并且通过残差通道，以允许模型以物理上有意义的方式解释它所能解释的一切，并且仍然在真实的上训练。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in part under the American Rescue Plan Act of 2021 (Public Law 117-2).This project aims to enable people to capture digital versions of real-world scenes more accurately, efficiently, and flexibly than is currently possible such that it can be a simple everyday task. In computer vision, capture of the real world is called reconstruction, and it is a core challenge that requires estimating the 3D shape, motion, object materials, and lighting within a scene. Successful reconstruction can provide spatial and geometric information of objects independent from lighting conditions to intelligent systems so that they can use the information for reasoning or creating applications of these objects from different viewing angles, e.g., in virtual and augmented reality. This project will scientifically investigate how to overcome the challenges of reconstruction by combining signals from different types of cameras in a way that is consistent with the physics of image formation. The project will integrate research and education by creating new interdisciplinary courses and promoting diversity from different outreach activities, e.g., supporting our K-12 AI4ALL local diversity effort, and attending inclusive teaching workshops at Brown’s Sheridan Center for Teaching and Learning. To help overcome the ill-posed problem of scene reconstruction from passive RGB cameras, this project has three areas of focus: 1) Investigate new camera systems that integrate multiple kinds of signals via physically based image formation models. Existing platforms handle typically one modality and frequency (visible light), but the project aims to combine visible light, time of flight, and event cameras to balance the negative effects of each camera and produce a signal of a quality that no individual camera could produce: high spatio-temporal resolution 3D video. 2) Investigate lighting and material decomposition via better capture, sampling, and reconstruction from heterogeneous omnidirectional cameras via new fast view synthesis methods adapted to represent incident illumination. This will use learned material priors from factorizations of physically based reflectance models that can exploit captured full and partial omnidirectional samples. 3) Investigate hybrid representations, optimization, and machine learning methods, including initialization based on reliable sparse sampling from depth sensors, via physically based self-supervised transforms to constrain optimization, and via residual error channels to allow the model to explain all that it can in a physically meaningful way and still train on real-world data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.