RI: Small: Uncovering Dynamics from Internet Imagery

虚拟现实和增强现实(VR/AR)技术有望使人们能够在舒适的家中和办公桌上以新的、令人兴奋的方式感知世界。在当前的3D VR可视化应用中，教育体验非常需要对真实世界环境的逼真描述。该项目将开发可扩展的算法，用于计算“活的3D模型”，这些模型可以表示场景中的人和车、喷泉中流动的水或咖啡馆外的椅子在不同日期被放置在不同地方等元素。为了克服专用捕捉的需求，该项目以公开可用的互联网照片集为目标，这些照片集具有必要的数据多样性，以推动大规模、具有成本效益的VR/AR内容生成。这项研究不仅支持VR/AR领域，还为其他广泛的应用提供了改进的分析方法，包括法医分析、文化遗产对话、城市规划、虚拟培训和教育，特别是在增强经济困难学生的社会学习体验方面。该项目将聚合照片集合的单个2D图像中的对象实例，以推断场景中整个对象类别的运动动力学，例如所有汽车或所有人。因此，该方法将在从未看到这些对象的运动的情况下推断运动动力学并对其建模，因为由于不受控制的众包捕获，每个对象实例通常仅有一个可用观测。推断的关键信息将是观察场景中动态场景元素的变化密度。新颖的场景表示将累积的动态存储在对象类场景占用图中，以及场景的对象类运动流，例如，行人在场景中移动到哪里以及他们如何在场景中移动的信息。开发的方法将为在野外联合恢复语义标签和3D几何图形的研究开辟新的令人兴奋的途径，这是目前3D计算机视觉中最具挑战性的问题之一。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Virtual- and augmented reality (VR/AR) technologies have the promise to enable new and exciting ways of perceiving the world from the comfort of our homes and desks. Among the current applications of 3D VR visualizations, obtaining realistic depictions of actual real-world environments is highly desired in educational experiences. This project will develop scalable algorithms for computing "living 3D models" that can represent elements such as people and cars moving around the scene, water flowing in fountains, or chairs outside cafes being placed in different places on different days. To overcome the need for dedicated capture, the project targets publicly available Internet photo collections, which have the requisite data diversity to drive large-scale, cost-effective VR/AR content generation. The research not only supports the field of VR/AR but also provides improved analysis methods for a broad range of other applications, including forensic analysis, cultural heritage conversation, city planning, virtual training, and education, with particularly potential impact in enhancing social study experiences for economically disadvantaged students. This project will aggregate object instances in the individual 2D images of the photo collection to infer the motion dynamics of the entire class of objects in the scene, e.g., all cars or all people. The method will thus infer and model the motion dynamics without ever seeing the motion of these objects, since there is typically only one observation per object instance available due to the uncontrolled, crowd-sourced capture. The key information for the inference will be the observation of the varying densities of the dynamic scene elements in the scene. The novel scene representation stores the accumulated dynamics in object class scene occupancy maps, as well as object class motion flows for the scene, e.g., the information where pedestrians move to in the scene and how they move within the scene. The developed methodology will open new and exciting avenues for research on jointly recovering semantic labels and 3D geometry in the wild, a task that is one of the currently most challenging problems in 3D computer vision.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.