Collaborative Research: CNS Core: Medium: Innovating Volumetric Video Streaming with Motion Forecasting, Intelligent Upsampling, and QoE Modeling

合作研究：CNS 核心：中：通过运动预测、智能上采样和 QoE 建模创新体积视频流

• 批准号: 2212296
• 负责人: Bo Han
• 金额: $ 40万
• 依托单位: George Mason University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2212296&HistoricalAwards=false
• 关键词: Collaborative; Research; CNS; Core; Medium

Volumetric video is an emerging type of multimedia content. Unlike traditional videos and 360-degree videos that are two-dimensional (2D), every frame in a volumetric video consists of a three-dimensional (3D) scene or object represented by a 3D model such as a point cloud or a polygon mesh. The 3D nature of volumetric video enables viewers to move with six degrees of freedom, leading to a truly immersive viewing experience. However, compared to conventional videos, streaming volumetric videos over the Internet faces unique challenges: a lack of precise motion tracking, extremely high bandwidth consumption, a lack of quality-of-experience (QoE) models, and complex interactions among network, computation, and motion. This project proposes a holistic research agenda addressing the above challenges through four core approaches: multi-dimension, multi-modality motion sensing and prediction; deep-learning-based content upsampling; comprehensive QoE modeling; and resource/motion adaptation. The project aims at demonstrable systems and networking research with a synergy among multimedia systems, networking, wireless sensing, machine learning, computer vision, and graphics. The research team will develop prototypes and evaluate them in real-world settings. Internet video streaming is playing a key role in today's world, especially during the COVID-19 pandemic. As a key enabler of immersive telepresence, volumetric content will become popular in the near future. The techniques developed from this project will significantly reduce network resource usage and improve the user experience for volumetric content delivery over the Internet. This will benefit both producers and consumers of volumetric videos, as well as boost their applications in various domains such as education, telehealth, manufacturing, and entertainment, ultimately leading to a high impact on societies and economies. The project will also offer new education components, and provide a platform for various Broadening Participation in Computing (BPC) activities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

立体视频是一种新兴的多媒体内容类型。与二维（2D）的传统视频和360度视频不同，体积视频中的每一帧都由三维（3D）场景或由3D模型（诸如点云或多边形网格）表示的对象组成。立体视频的3D特性使观众能够以六个自由度移动，从而带来真正身临其境的观看体验。然而，与传统视频相比，通过互联网流式传输体积视频面临着独特的挑战：缺乏精确的运动跟踪，极高的带宽消耗，缺乏体验质量（QoE）模型，以及网络，计算和运动之间的复杂交互。该项目提出了一个整体研究议程，通过四种核心方法解决上述挑战：多维，多模态运动感知和预测;基于深度学习的内容上采样;全面的QoE建模;和资源/运动自适应。该项目旨在通过多媒体系统，网络，无线传感，机器学习，计算机视觉和图形之间的协同作用进行可演示的系统和网络研究。研究团队将开发原型，并在现实世界中进行评估。互联网视频流在当今世界发挥着关键作用，尤其是在COVID-19大流行期间。作为沉浸式远程呈现的关键推动因素，体积内容将在不久的将来变得流行。从这个项目开发的技术将大大减少网络资源的使用，并改善用户的经验，通过互联网的批量内容交付。这将有利于立体视频的制作者和消费者，并促进其在教育、远程医疗、制造业和娱乐等各个领域的应用，最终对社会和经济产生重大影响。该项目还将提供新的教育内容，并为各种扩大参与计算（BPC）活动提供一个平台。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Demystifying Web-based Mobile Extended Reality Accelerated by WebAssembly

• DOI: 10.1145/3618257.3624833
• 发表时间: 2023-10
• 期刊: Proceedings of the 2023 ACM on Internet Measurement Conference
• 作者: Kaiyan Liu;Nan Wu;Bo Han

MetaStream: Live Volumetric Content Capture, Creation, Delivery, and Rendering in Real Time

• DOI: 10.1145/3570361.3592530
• 发表时间: 2023-10
• 期刊: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking
• 作者: Yongjie Guan;Xueyu Hou;Nan Wu;Bo Han;Tao Han

Environment-driven mmWave Beamforming for Multi-user Immersive Applications

• DOI: 10.1145/3615452.3617945
• 发表时间: 2023-10
• 期刊: Proceedings of the 1st ACM Workshop on Mobile Immersive Computing, Networking, and Systems
• 作者: Ahmad Kamari;Nan Wu;Hemant Kumar Narsani;Bo Han;Parth Pathak

Toward Next-generation Volumetric Video Streaming with Neural-based Content Representations

• DOI: 10.1145/3615452.3617938
• 发表时间: 2023-10
• 期刊: Proceedings of the 1st ACM Workshop on Mobile Immersive Computing, Networking, and Systems

Enriching Telepresence with Semantic-driven Holographic Communication

• DOI: 10.1145/3626111.3628184
• 发表时间: 2023-11
• 期刊: Proceedings of the 22nd ACM Workshop on Hot Topics in Networks
• 作者: Ruizhi Cheng;Kaiyan Liu;Nan Wu;Bo Han