RI: Small: Uncovering Dynamics from Internet Imagery

虚拟现实和增强现实（VR/AR）技术有望实现从我们舒适的家中和办公桌上感知世界的新的令人兴奋的方式。在3D VR可视化的当前应用中，在教育体验中非常期望获得实际现实世界环境的逼真的演示。该项目将开发用于计算“活3D模型”的可扩展算法，这些模型可以表示在场景中移动的人和汽车，喷泉中流动的水，或咖啡馆外的椅子在不同的日子被放置在不同的地方等元素。为了克服对专用捕获的需求，该项目针对公开的互联网照片集，这些照片集具有必要的数据多样性，可以驱动大规模，具有成本效益的VR/AR内容生成。该研究不仅支持VR/AR领域，还为广泛的其他应用提供了改进的分析方法，包括法医分析，文化遗产对话，城市规划，虚拟培训和教育，特别是对增强经济困难学生的社会学习经验具有潜在影响。该项目将在照片集的各个2D图像中聚合对象实例，以推断场景中整个对象类的运动动态，例如，所有的汽车或所有的人。因此，该方法将推断和建模运动动态，而无需看到这些对象的运动，因为由于不受控制的众包捕获，通常每个对象实例只有一个观察结果可用。用于推断的关键信息将是对场景中动态场景元素的变化密度的观察。新颖的场景表示将累积的动态存储在对象类场景占用图中，以及存储场景的对象类运动流，例如，行人在场景中移动的位置以及他们如何在场景中移动的信息。开发的方法将为在野外联合恢复语义标签和3D几何形状的研究开辟新的令人兴奋的途径，这是目前3D计算机视觉中最具挑战性的问题之一。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Virtual- and augmented reality (VR/AR) technologies have the promise to enable new and exciting ways of perceiving the world from the comfort of our homes and desks. Among the current applications of 3D VR visualizations, obtaining realistic depictions of actual real-world environments is highly desired in educational experiences. This project will develop scalable algorithms for computing "living 3D models" that can represent elements such as people and cars moving around the scene, water flowing in fountains, or chairs outside cafes being placed in different places on different days. To overcome the need for dedicated capture, the project targets publicly available Internet photo collections, which have the requisite data diversity to drive large-scale, cost-effective VR/AR content generation. The research not only supports the field of VR/AR but also provides improved analysis methods for a broad range of other applications, including forensic analysis, cultural heritage conversation, city planning, virtual training, and education, with particularly potential impact in enhancing social study experiences for economically disadvantaged students. This project will aggregate object instances in the individual 2D images of the photo collection to infer the motion dynamics of the entire class of objects in the scene, e.g., all cars or all people. The method will thus infer and model the motion dynamics without ever seeing the motion of these objects, since there is typically only one observation per object instance available due to the uncontrolled, crowd-sourced capture. The key information for the inference will be the observation of the varying densities of the dynamic scene elements in the scene. The novel scene representation stores the accumulated dynamics in object class scene occupancy maps, as well as object class motion flows for the scene, e.g., the information where pedestrians move to in the scene and how they move within the scene. The developed methodology will open new and exciting avenues for research on jointly recovering semantic labels and 3D geometry in the wild, a task that is one of the currently most challenging problems in 3D computer vision.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.