CAREER: RUI: CasualVR: Casual Capture of Dynamic Virtual Reality Environments

该奖项全部或部分由《2021年美国救援计划法案》（公法117-2）资助。对远程连接、学习和共享的新方式的需求从未像现在这样迫切和重要。虽然目前大多数数字媒体都是平面的二维图像和视频，但虚拟现实技术将允许共享一个场景的完整三维表示，以实现真正的协作体验。有了虚拟现实技术，学生们可以虚拟地站在古代金字塔的入口处，在著名的博物馆里看画，或者探索雨林。为了实现这个高水平的目标，这个项目的目的是让任何拥有智能手机的人都能轻松地捕捉和分享他们周围环境的高度沉浸式虚拟现实体验。共享高质量虚拟现实体验的普及和消费者可访问技术的发展，将能够创造更有影响力和持久的学习体验，通过使人们更有说服力地体验他人的生活来支持同理心的发展，并允许更忠实地记录重要的经历，否则可能会丢失。这个项目的研究目标将与若干教育和外联活动结合起来。研究者将与生物学和地理学的学生和教师组成的跨学科团队合作，开发关于加州生物多样性的沉浸式教育内容，并将为加州学生实施一个外展研讨会，向他们介绍三维重建和视图合成背后的概念，并让他们使用开发的工具创建自己的虚拟现实体验。作为研究活动的一部分生成的数据集将被公开，作为其他研究人员和应用程序开发人员的重要资源。尽管最近的发展势头迅猛，但在从随意捕获的视频中重建虚拟现实体验、处理动态场景和渲染速度方面仍然存在关键问题。该项目旨在通过探索捕捉和重建动态物理环境的新技术来缩小这些差距。为了解决内容捕获和重建方面的挑战，该项目将探索使用神经隐式场景表示从随意捕获的视频中自动自校准和动态场景重建的新方法。对于更复杂和动态的场景来说，这是一个具有挑战性的任务，在那里很难找到一个单一的过渡（从场景到场景），在整个体积中看起来是无缝的。为了更快地捕捉和更好地处理动态场景，该项目将开发深度学习技术，自动将消费者全景视频转换为动态和沉浸式多层表示。为了解决存储效率和快速渲染问题，该项目将研究基于图像的神经光场捕获和重建。该项目还将生成新的数据集，用于训练和评估动态场景中的沉浸式视图合成方法，并开发新的测量/指标，用于评估生成的沉浸式内容。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).The need for compelling new ways to connect, learn, and share remotely has never been more immediate and vital for society. While most digital media today consist of flat, two-dimensional images and video, virtual reality technology will allow for sharing a complete three-dimensional representation of a scene, for a truly collaborative experience. With virtual reality technology, schoolchildren could virtually stand at the entrance of an ancient pyramid, look at paintings in a renowned museum, or explore a rainforest. Towards this high-level objective, the aim of this project is to enable anyone with a smartphone to easily capture and share a highly immersive virtual reality experience of their surroundings. The development of widespread and consumer-accessible technology for sharing high-quality virtual reality experiences would enable the creation of more impactful and lasting learning experiences, support the development of empathy by enabling people to experience the lives of others more convincingly, and allow for more faithfully recordings of important experiences that might otherwise be lost. The research objectives of this project will be integrated with several education and outreach activities. The investigator will work with an interdisciplinary team of students and faculty in biology and geography to develop immersive educational content about biodiversity in California and will implement an outreach workshop for California students to introduce them to the concepts behind three-dimensional reconstruction and view synthesis and have them create their own virtual reality experiences using the tools developed. Datasets generated as part of the research activities will be made public, serving as a great resource for other researchers and application developers.Despite a groundswell of recent developments, critical issues remain in terms of reconstructing virtual reality experiences from casually captured video, handling dynamic scenes, and rendering speed. This project aims to close these gaps by exploring novel techniques for capture and reconstruction of dynamic physical environments. To address challenges in content capture and reconstruction, the project will explore new methods for automatic self-calibration and dynamic scene reconstruction from casually captured video using a neural implicit scene representation. This becomes a challenging task for more complex and dynamic scenes, where it might be tough to find a single transition (from scene to scene) which appears seamless across the entire volume. For faster capture and better handling of dynamic scenes, the project will develop deep learning techniques to automatically convert consumer panoramic video into a dynamic and immersive multi-layer representation. Addressing storage efficiency and fast rendering, the project will investigate image-based capture and reconstruction of neural light fields. The project will also produce new datasets for training and evaluating immersive view synthesis methods in dynamic scenes and develop new measures/metrics for the evaluation of the generated immersive content.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.