CAREER: System Research to Enable Practical Immersive Streaming: From 360-Degree Towards Volumetric Video Delivery

沉浸式视频技术允许用户自由探索远程或虚拟环境。例如，使用360度视频，用户可以从任何方向查看场景；使用体积视频，用户不仅可以控制查看方向，还可以控制摄像头位置。这种高度身临其境的内容在娱乐、医疗、教育、制造和电子商务等领域都有应用。然而，目前的视频流基础设施不能完全支持这些新兴格式的高带宽、低延迟和高存储要求。该项目旨在通过提出沉浸式视频流的新的、高效的表示来解决在高效传输和存储沉浸式视频流方面的挑战。这些表示既可以适应用户的观看行为，也可以在实时流应用所需的低延迟下生成。如果成功，拟议的研究将实现比当前系统更高质量的沉浸式流媒体，进一步实现有用的沉浸式流媒体体验。本项目研究提高两个特定沉浸式流媒体应用程序的效率的技术。对于实时360度视频，该项目将建立一个系统来实时生成焦点区域投影。选择这些焦点区域投影以使高质量焦点区域与预测的用户视图对准。使用附近边缘或云服务器上的图形处理单元可实现低延迟生成。对于体积视频流，该项目的目标是创建存储和带宽高效的视频表示。该表示包括体积内选定点集的视频的焦点区域版本以及覆盖非遮挡像素的补丁，从而允许将高质量视频传送给场景中任何位置的用户。所提出的系统还使用超文本传输协议版本2(HTTP/2)传输方法来支持该表示的带宽高效传递。为了更准确地衡量用户对沉浸式视频流的真实体验，该项目还将创建沉浸式视频流数据集，并调查新的质量指标。这一新的衡量标准将使用新的方法将用户感知的质量与由于编码和网络传输而引入的视觉伪像关联起来。由于该项目而产生的构件，包括出版物、代码和数据集，将在http://www.cs.binghamton.edu/~yaoliu/ImmersiveStreaming/.上公开提供这些文物将在项目完成后至少保留五年。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Immersive video technologies allow users to freely explore remote or virtual environments. For example, with 360-degree videos, users can view the scene from any orientation; with volumetric videos, users can control not only the view orientation but also the camera position. Such highly immersive content has applications in entertainment, medicine, education, manufacturing, and e-commerce, to name just a few areas. However, current video streaming infrastructure cannot fully support these emerging formats’ high bandwidth, low latency, and high storage requirements. This project aims to address challenges in efficient transmission and storage of immersive video streams by proposing new, efficient representations of this content. These representations can both be adapted to fit users’ viewing behaviors and can be generated at the low-latencies required for real-time streaming applications. If successful, the proposed research will enable higher quality immersive streaming than is possible with current systems, further enabling useful immersive streaming experiences. This project investigates techniques for improving efficiency of two specific immersive streaming applications. For real-time 360-degree video, the project will build a system to generate area-of-focus projections in real-time. These area-of-focus projections are selected to align the high-quality focus area with a predicted user view. Low-latency generation is achieved using graphics processing units at nearby edge or cloud servers. For volumetric video streaming, this project aims to create both a storage- and bandwidth-efficient representation of the video. The representation consists of both area-of-focus versions of the video at a selected set of points within the volume as well as patches to cover dis-occluded pixels, allowing high-quality video to be delivered to users positioned anywhere in the scene. The proposed system further uses a Hypertext Transfer Protocol Version 2 (HTTP/2) transmission approach to support bandwidth-efficient delivery of this representation. To more precisely measure the user’s true experience of immersive streams, this project will also create immersive video streaming datasets and investigate a new quality metric. This new metric will use novel approaches to correlate user-perceived qualities with visual artifacts introduced due to encoding and network transmission. Artifacts produced as a result of this project, including publications, code, and datasets, will be made publicly available at http://www.cs.binghamton.edu/~yaoliu/ImmersiveStreaming/. These artifacts will be maintained for at least five years after completion of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.