CAREER: Spatial Awareness for Machine Perception

职业：机器感知的空间意识

• 批准号: 2046910
• 负责人: Carl Vondrick
• 金额: $ 55万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2046910&HistoricalAwards=false
• 关键词: CAREER; Spatial; Awareness; Machine; Perception

For many applications in health, security, and robotics, the capability for computers to track objects in video is critical. From autonomous vehicles that reliably avoid obstacles to security cameras that swiftly respond during emergencies, artificial perception systems need to remain accurate during poor visibility situations, such as with cluttered rooms, dark streets, or occluded objects. This integrated research and education project aims to develop machines that are able to spatially sense their visual surroundings, even when the surroundings have low visibility with significant obstructions. On multiple levels, the investigators will furthermore use their integrated approach to advance education in artificial intelligence, such as outreach activities for K-12 students and streamlined curriculums for both undergraduate and graduate education in computer vision.The research program introduces a framework that tightly integrates geometry, motion, and acoustics in order to learn rich spatial representations of natural scenes from unlabeled visual data. The research team will study how learning from the incidental structures of unlabeled videos will cause rich spatial representations to emerge without human supervision. The framework will leverage the propagation of sound through objects to learn visual tracking under occlusion. In other cases, the project will use scene geometry to transfer incidental supervision between camera views in order to learn spatial memory models. A key advantage of the approach is the efficiency to operate on videos that span both large physical spaces and long temporal horizons. Motivated by implicit surfaces in mathematics and graphics, the algorithm will analytically represents videos as continuous functions, which are compact and robustly embed video dynamics in 3D space.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

对于健康、安全和机器人领域的许多应用来说，计算机跟踪视频中的物体的能力至关重要。从可靠地避开障碍物的自动驾驶汽车，到在紧急情况下迅速做出反应的安全摄像头，人工感知系统需要在能见度较低的情况下保持准确，比如杂乱的房间、黑暗的街道或遮挡的物体。这个综合研究和教育项目旨在开发能够在空间上感知其视觉环境的机器，即使周围环境能见度低且有明显障碍物。在多个层面上，研究人员将进一步使用他们的综合方法来推进人工智能教育，例如为K-12学生提供外展活动，并简化计算机视觉本科和研究生教育的课程。该研究项目引入了一个框架，将几何、运动和声学紧密结合在一起，以便从未标记的视觉数据中学习丰富的自然场景空间表示。研究小组将研究如何从未标记视频的附带结构中学习，从而在没有人类监督的情况下产生丰富的空间表征。该框架将利用声音在物体中的传播来学习遮挡下的视觉跟踪。在其他情况下，该项目将使用场景几何来转移相机视图之间的偶然监督，以学习空间记忆模型。这种方法的一个关键优势是，在跨越大的物理空间和长时间视界的视频上操作的效率。该算法以数学和图形中的隐式曲面为动力，将视频解析表示为连续函数，紧凑且鲁棒地嵌入了三维空间中的视频动态。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Revealing Occlusions with 4D Neural Fields

• DOI: 10.1109/cvpr52688.2022.00302
• 发表时间: 2022-04
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Basile Van Hoorick;Purva Tendulkar;Dídac Surís;Dennis Park;Simon Stent;Carl Vondrick

Tracking Through Containers and Occluders in the Wild

• DOI: 10.1109/cvpr52729.2023.01326
• 发表时间: 2023-05
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Basile Van Hoorick;P. Tokmakov;Simon Stent;Jie Li;Carl Vondrick

What You Can Reconstruct from a Shadow

你可以从阴影中重建什么

• 发表时间: 2023
• 期刊: IEEE Conference on Computer Vision and Pattern Recognition
• 作者: Ruoshi Liu;Sachit Menon;Chengzhi Mao;Dennis Park;Simon Stent;Carl Vondrick
• 通讯作者: Carl Vondrick

Humans as Light Bulbs: 3D Human Reconstruction from Thermal Reflection

• DOI: 10.1109/cvpr52729.2023.01206
• 发表时间: 2023-05
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Ruoshi Liu;Carl Vondrick

Real-Time Neural Voice Camouflage

• 发表时间: 2021-12
• 期刊: ArXiv
• 作者: Mia Chiquier;Chengzhi Mao;Carl Vondrick