CAREER: System Research to Enable Practical Immersive Streaming: From 360-Degree Towards Volumetric Video Delivery

沉浸式视频技术允许用户自由地探索远程或虚拟环境。例如，使用360度视频，用户可以从任何方向观看场景；使用体积视频，用户不仅可以控制观看方向，还可以控制摄像机的位置。这种高度沉浸式的内容在娱乐、医疗、教育、制造业和电子商务等领域都有应用。然而，目前的视频流基础设施还不能完全支持这些新兴格式的高带宽、低延迟和高存储要求。该项目旨在通过提出新的、高效的内容表示来解决高效传输和存储沉浸式视频流的挑战。这些表示既可以适应用户的观看行为，也可以在实时流媒体应用程序所需的低延迟下生成。如果成功，该研究将实现比当前系统更高质量的沉浸式流媒体，进一步实现有用的沉浸式流媒体体验。本项目研究了提高两种特定沉浸式流媒体应用程序效率的技术。对于实时360度视频，该项目将建立一个实时生成焦点区域投影的系统。选择这些焦点区域投影来将高质量的焦点区域与预测的用户视图对齐。使用附近边缘服务器或云服务器上的图形处理单元实现低延迟生成。对于容量视频流，该项目旨在创建存储和带宽效率高的视频表示。该表示包括视频在体积内选定点的聚焦区域版本，以及覆盖未遮挡像素的补丁，从而允许将高质量视频传送给场景中任何位置的用户。提出的系统进一步使用超文本传输协议版本2 （HTTP/2）传输方法来支持这种表示的带宽高效交付。为了更精确地测量用户对沉浸式视频流的真实体验，该项目还将创建沉浸式视频流数据集，并研究一种新的质量度量标准。这种新的度量将使用新颖的方法将用户感知的质量与由于编码和网络传输而引入的视觉伪影联系起来。作为这个项目的结果产生的工件，包括出版物、代码和数据集，将在http://www.cs.binghamton.edu/~yaoliu/ImmersiveStreaming/上公开提供。这些文物将在项目完成后至少保存五年。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Immersive video technologies allow users to freely explore remote or virtual environments. For example, with 360-degree videos, users can view the scene from any orientation; with volumetric videos, users can control not only the view orientation but also the camera position. Such highly immersive content has applications in entertainment, medicine, education, manufacturing, and e-commerce, to name just a few areas. However, current video streaming infrastructure cannot fully support these emerging formats’ high bandwidth, low latency, and high storage requirements. This project aims to address challenges in efficient transmission and storage of immersive video streams by proposing new, efficient representations of this content. These representations can both be adapted to fit users’ viewing behaviors and can be generated at the low-latencies required for real-time streaming applications. If successful, the proposed research will enable higher quality immersive streaming than is possible with current systems, further enabling useful immersive streaming experiences. This project investigates techniques for improving efficiency of two specific immersive streaming applications. For real-time 360-degree video, the project will build a system to generate area-of-focus projections in real-time. These area-of-focus projections are selected to align the high-quality focus area with a predicted user view. Low-latency generation is achieved using graphics processing units at nearby edge or cloud servers. For volumetric video streaming, this project aims to create both a storage- and bandwidth-efficient representation of the video. The representation consists of both area-of-focus versions of the video at a selected set of points within the volume as well as patches to cover dis-occluded pixels, allowing high-quality video to be delivered to users positioned anywhere in the scene. The proposed system further uses a Hypertext Transfer Protocol Version 2 (HTTP/2) transmission approach to support bandwidth-efficient delivery of this representation. To more precisely measure the user’s true experience of immersive streams, this project will also create immersive video streaming datasets and investigate a new quality metric. This new metric will use novel approaches to correlate user-perceived qualities with visual artifacts introduced due to encoding and network transmission. Artifacts produced as a result of this project, including publications, code, and datasets, will be made publicly available at http://www.cs.binghamton.edu/~yaoliu/ImmersiveStreaming/. These artifacts will be maintained for at least five years after completion of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.