Collaborative Research: CNS Core: Medium: Innovating Volumetric Video Streaming with Motion Forecasting, Intelligent Upsampling, and QoE Modeling

体积视频是一种新兴的多媒体内容类型。与二维 (2D) 的传统视频和 360 度视频不同，体积视频中的每一帧都包含由 3D 模型（例如点云或多边形网格）表示的三维 (3D) 场景或对象。体积视频的 3D 特性使观看者能够以六个自由度移动，从而带来真正身临其境的观看体验。然而，与传统视频相比，通过互联网传输体积视频面临着独特的挑战：缺乏精确的运动跟踪、极高的带宽消耗、缺乏体验质量（QoE）模型以及网络、计算和运动之间复杂的交互。该项目提出了一个整体研究议程，通过四个核心方法解决上述挑战：多维度、多模态运动传感和预测；基于深度学习的内容上采样；全面的 QoE 建模；和资源/运动适应。该项目旨在通过多媒体系统、网络、无线传感、机器学习、计算机视觉和图形之间的协同作用进行可演示的系统和网络研究。研究团队将开发原型并在现实环境中对其进行评估。互联网视频流在当今世界发挥着关键作用，尤其是在新冠肺炎 (COVID-19) 大流行期间。作为沉浸式远程呈现的关键推动者，立体内容将在不久的将来变得流行。该项目开发的技术将显着减少网络资源的使用，并改善通过互联网传输大量内容的用户体验。这将使体积视频的制作者和消费者受益，并促进其在教育、远程医疗、制造和娱乐等各个领域的应用，最终对社会和经济产生重大影响。该项目还将提供新的教育内容，并为各种扩大计算参与 (BPC) 活动提供平台。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，认为值得支持。

Volumetric video is an emerging type of multimedia content. Unlike traditional videos and 360-degree videos that are two-dimensional (2D), every frame in a volumetric video consists of a three-dimensional (3D) scene or object represented by a 3D model such as a point cloud or a polygon mesh. The 3D nature of volumetric video enables viewers to move with six degrees of freedom, leading to a truly immersive viewing experience. However, compared to conventional videos, streaming volumetric videos over the Internet faces unique challenges: a lack of precise motion tracking, extremely high bandwidth consumption, a lack of quality-of-experience (QoE) models, and complex interactions among network, computation, and motion. This project proposes a holistic research agenda addressing the above challenges through four core approaches: multi-dimension, multi-modality motion sensing and prediction; deep-learning-based content upsampling; comprehensive QoE modeling; and resource/motion adaptation. The project aims at demonstrable systems and networking research with a synergy among multimedia systems, networking, wireless sensing, machine learning, computer vision, and graphics. The research team will develop prototypes and evaluate them in real-world settings. Internet video streaming is playing a key role in today's world, especially during the COVID-19 pandemic. As a key enabler of immersive telepresence, volumetric content will become popular in the near future. The techniques developed from this project will significantly reduce network resource usage and improve the user experience for volumetric content delivery over the Internet. This will benefit both producers and consumers of volumetric videos, as well as boost their applications in various domains such as education, telehealth, manufacturing, and entertainment, ultimately leading to a high impact on societies and economies. The project will also offer new education components, and provide a platform for various Broadening Participation in Computing (BPC) activities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.