CAREER: RUI: CasualVR: Casual Capture of Dynamic Virtual Reality Environments

职业：RUI：CasualVR：动态虚拟现实环境的休闲捕捉

• 批准号: 2144822
• 负责人: Jonathan Ventura
• 金额: $ 42.58万
• 依托单位: California Polytechnic State University Foundation
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-07-01 至 2027-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2144822&HistoricalAwards=false
• 关键词: CAREER; RUI; CasualVR; Casual; Capture

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).The need for compelling new ways to connect, learn, and share remotely has never been more immediate and vital for society. While most digital media today consist of flat, two-dimensional images and video, virtual reality technology will allow for sharing a complete three-dimensional representation of a scene, for a truly collaborative experience. With virtual reality technology, schoolchildren could virtually stand at the entrance of an ancient pyramid, look at paintings in a renowned museum, or explore a rainforest. Towards this high-level objective, the aim of this project is to enable anyone with a smartphone to easily capture and share a highly immersive virtual reality experience of their surroundings. The development of widespread and consumer-accessible technology for sharing high-quality virtual reality experiences would enable the creation of more impactful and lasting learning experiences, support the development of empathy by enabling people to experience the lives of others more convincingly, and allow for more faithfully recordings of important experiences that might otherwise be lost. The research objectives of this project will be integrated with several education and outreach activities. The investigator will work with an interdisciplinary team of students and faculty in biology and geography to develop immersive educational content about biodiversity in California and will implement an outreach workshop for California students to introduce them to the concepts behind three-dimensional reconstruction and view synthesis and have them create their own virtual reality experiences using the tools developed. Datasets generated as part of the research activities will be made public, serving as a great resource for other researchers and application developers.Despite a groundswell of recent developments, critical issues remain in terms of reconstructing virtual reality experiences from casually captured video, handling dynamic scenes, and rendering speed. This project aims to close these gaps by exploring novel techniques for capture and reconstruction of dynamic physical environments. To address challenges in content capture and reconstruction, the project will explore new methods for automatic self-calibration and dynamic scene reconstruction from casually captured video using a neural implicit scene representation. This becomes a challenging task for more complex and dynamic scenes, where it might be tough to find a single transition (from scene to scene) which appears seamless across the entire volume. For faster capture and better handling of dynamic scenes, the project will develop deep learning techniques to automatically convert consumer panoramic video into a dynamic and immersive multi-layer representation. Addressing storage efficiency and fast rendering, the project will investigate image-based capture and reconstruction of neural light fields. The project will also produce new datasets for training and evaluating immersive view synthesis methods in dynamic scenes and develop new measures/metrics for the evaluation of the generated immersive content.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该奖项全部或部分由2021年美国救援计划法案（公法117-2）资助。对远程连接，学习和共享的新方式的需求从未如此紧迫和重要。 虽然今天的大多数数字媒体都是平面的二维图像和视频，但虚拟现实技术将允许共享场景的完整三维表示，以实现真正的协作体验。 通过虚拟现实技术，学生们可以虚拟地站在古代金字塔的入口处，在著名的博物馆里欣赏画作，或者探索热带雨林。 为了实现这一高层次的目标，该项目的目的是使任何拥有智能手机的人都能够轻松地捕捉和分享他们周围环境的高度沉浸式虚拟现实体验。 开发广泛和消费者可访问的技术来分享高质量的虚拟现实体验，将能够创造更有影响力和持久的学习体验，通过使人们更有说服力地体验他人的生活来支持同理心的发展，并允许更忠实地记录重要的经验，否则可能会丢失。 该项目的研究目标将与若干教育和外联活动结合起来。研究人员将与生物学和地理学的学生和教师组成的跨学科团队合作，开发关于加州生物多样性的沉浸式教育内容，并将为加州学生举办外展研讨会，向他们介绍三维重建和视图合成背后的概念，并让他们使用开发的工具创建自己的虚拟现实体验。作为研究活动的一部分生成的数据集将被公开，为其他研究人员和应用程序开发人员提供重要资源。尽管最近的发展风起云涌，但在从随意捕获的视频重建虚拟现实体验，处理动态场景和渲染速度方面仍然存在关键问题。 该项目旨在通过探索捕捉和重建动态物理环境的新技术来缩小这些差距。为了应对内容捕获和重建方面的挑战，该项目将探索使用神经隐式场景表示从随意捕获的视频中自动自校准和动态场景重建的新方法。对于更复杂和动态的场景，这成为一项具有挑战性的任务，在这些场景中，可能很坚韧找到一个在整个体积中无缝出现的单一过渡（从场景到场景）。为了更快地捕捉和更好地处理动态场景，该项目将开发深度学习技术，以自动将消费者全景视频转换为动态和沉浸式的多层表示。 为了解决存储效率和快速渲染问题，该项目将研究基于图像的神经光场捕获和重建。 该项目还将产生新的数据集，用于培训和评估动态场景中的沉浸式视图合成方法，并为评估生成的沉浸式内容开发新的措施/指标。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Localization and tracking of stationary users for augmented reality

• DOI: 10.1007/s00371-023-02777-2
• 发表时间: 2023-02-02
• 期刊: VISUAL COMPUTER
• 影响因子: 3.5
• 作者: Baker, Lewis;Ventura, Jonathan;Zollmann, Stefanie
• 通讯作者: Zollmann, Stefanie

PanoSynthVR: Toward Light-weight 360-Degree View Synthesis from a Single Panoramic Input

• DOI: 10.1109/ismar55827.2022.00075
• 发表时间: 2022-10
• 期刊: 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)
• 作者: John Waidhofer;Richa Gadgil;Anthony Dickson;S. Zollmann;Jonathan Ventura

VRVideos: A flexible pipeline for Virtual Reality Video Creation

VRVideos：虚拟现实视频创建的灵活管道

• DOI: 10.1109/aivr56993.2022.00039
• 发表时间: 2022
• 期刊: 2022 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR
• 作者: Dickson, Anthony;Shanks, Jeremy;Ventura, Jonathan;Knott, Alistair;Zollmann, Stefanie
• 通讯作者: Zollmann, Stefanie