CAREER: System Research to Enable Practical Immersive Streaming: From 360-Degree Towards Volumetric Video Delivery

职业：实现实用沉浸式流媒体的系统研究：从 360 度到立体视频传输

• 批准号: 2200048
• 负责人: Yao Liu
• 金额: $ 48.62万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2025-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2200048&HistoricalAwards=false
• 关键词: CAREER; System; Research; Enable; Practical

Immersive video technologies allow users to freely explore remote or virtual environments. For example, with 360-degree videos, users can view the scene from any orientation; with volumetric videos, users can control not only the view orientation but also the camera position. Such highly immersive content has applications in entertainment, medicine, education, manufacturing, and e-commerce, to name just a few areas. However, current video streaming infrastructure cannot fully support these emerging formats’ high bandwidth, low latency, and high storage requirements. This project aims to address challenges in efficient transmission and storage of immersive video streams by proposing new, efficient representations of this content. These representations can both be adapted to fit users’ viewing behaviors and can be generated at the low-latencies required for real-time streaming applications. If successful, the proposed research will enable higher quality immersive streaming than is possible with current systems, further enabling useful immersive streaming experiences. This project investigates techniques for improving efficiency of two specific immersive streaming applications. For real-time 360-degree video, the project will build a system to generate area-of-focus projections in real-time. These area-of-focus projections are selected to align the high-quality focus area with a predicted user view. Low-latency generation is achieved using graphics processing units at nearby edge or cloud servers. For volumetric video streaming, this project aims to create both a storage- and bandwidth-efficient representation of the video. The representation consists of both area-of-focus versions of the video at a selected set of points within the volume as well as patches to cover dis-occluded pixels, allowing high-quality video to be delivered to users positioned anywhere in the scene. The proposed system further uses a Hypertext Transfer Protocol Version 2 (HTTP/2) transmission approach to support bandwidth-efficient delivery of this representation. To more precisely measure the user’s true experience of immersive streams, this project will also create immersive video streaming datasets and investigate a new quality metric. This new metric will use novel approaches to correlate user-perceived qualities with visual artifacts introduced due to encoding and network transmission. Artifacts produced as a result of this project, including publications, code, and datasets, will be made publicly available at http://www.cs.binghamton.edu/~yaoliu/ImmersiveStreaming/. These artifacts will be maintained for at least five years after completion of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

沉浸式视频技术允许用户自由探索远程或虚拟环境。例如，对于360度视频，用户可以从任何方向查看场景;对于体积视频，用户不仅可以控制视图方向，还可以控制相机位置。这种高度沉浸式的内容在娱乐、医学、教育、制造业和电子商务等领域都有应用。然而，当前的视频流基础设施不能完全支持这些新兴格式的高带宽、低延迟和高存储要求。该项目旨在通过提出这种内容的新的高效表示来解决沉浸式视频流的高效传输和存储方面的挑战。这些表示可以适应用户的观看行为，并且可以以实时流应用所需的低延迟生成。如果成功，拟议的研究将实现比当前系统更高质量的沉浸式流媒体，进一步实现有用的沉浸式流媒体体验。该项目研究用于提高两个特定沉浸式流媒体应用程序的效率的技术。对于实时360度视频，该项目将建立一个系统来实时生成聚焦投影。选择这些聚焦区域投影以将高质量聚焦区域与预测的用户视图对准。低延迟生成是使用附近边缘或云服务器上的图形处理单元实现的。对于体积视频流，该项目旨在创建一个存储和带宽效率的视频表示。该表示包括在体积内选定的一组点处的视频的聚焦区域版本以及覆盖被遮挡像素的补丁，从而允许将高质量视频传递给位于场景中任何位置的用户。所提出的系统还使用超文本传输协议版本2（HTTP/2）传输方法来支持该表示的带宽高效递送。为了更精确地衡量用户对沉浸式流的真实体验，该项目还将创建沉浸式视频流数据集并研究新的质量指标。这个新的度量标准将使用新的方法来将用户感知的质量与由于编码和网络传输而引入的视觉伪影相关联。作为该项目的结果产生的工件，包括出版物，代码和数据集，将在http://www.cs.binghamton.edu/~yaoliu/ImmersiveStreaming/上公开。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Exploring Spherical Autoencoder for Spherical Video Content Processing

• DOI: 10.1145/3503161.3548364
• 发表时间: 2022-10
• 期刊: Proceedings of the 30th ACM International Conference on Multimedia
• 作者: Jin Zhou;Na Li;Yao Liu-;Shuochao Yao;Songqing Chen

EVASR: Edge-Based Video Delivery with Salience-Aware Super-Resolution

EVASR：具有显着性感知超分辨率的基于边缘的视频传输

• DOI: 10.1145/3587819.3590967
• 发表时间: 2023
• 期刊: Proceedings of the 14th Conference on ACM Multimedia Systems
• 作者: Li, Na;Liu, Yao
• 通讯作者: Liu, Yao

Security-Preserving Live 3D Video Surveillance

• DOI: 10.1145/3587819.3590975
• 发表时间: 2023-06
• 期刊: Proceedings of the 14th Conference on ACM Multimedia Systems
• 作者: Zhongze Tang;Huy Phan;Xianglong Feng;Bo Yuan;Yao Liu;Sheng Wei

VertexShuffle-Based Spherical Super-Resolution for 360-Degree Videos

• DOI: 10.1145/3649315
• 发表时间: 2024-02
• 期刊: ACM Transactions on Multimedia Computing, Communications and Applications
• 作者: Na Li;Yao Liu

VQBA: Visual-Quality-Driven Bit Allocation for Low-Latency Point Cloud Streaming

VQBA：低延迟点云流的视觉质量驱动的比特分配

• DOI: 10.1145/3581783.3612486
• 发表时间: 2023
• 期刊: ACM
• 作者: Wang, Shuoqian;Zhu, Mufeng;Li, Na;Xiao, Mengbai;Liu, Yao
• 通讯作者: Liu, Yao