NeTS: Small: Dynamic Predictive Streaming of 360 Degree Video

NeTS：小型：360 度视频的动态预测流

• 批准号: 1816500
• 负责人: Yong Liu
• 金额: $ 49.99万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1816500&HistoricalAwards=false
• 关键词: NeTS; Small; Dynamic; Predictive; Streaming

Virtual Reality (VR) and Augmented Reality (AR) applications are projected to be the next wave of "Killer Apps" in the future Internet. VR/AR applications facilitate vivid immersive virtual and augmented reality experience and create tremendous new opportunities in many domains, including education, business, healthcare, and entertainment, etc. Many VR/AR applications involve streaming of 360-degree video scenes. Compared with the traditional video streaming, 360-degree video streaming requires much higher network bandwidth and much lower packet delivery latency, and user's quality of experience is highly sensitive to the dynamics in both network environment and user viewing behaviors. Addressing these unique challenges, this project will develop novel 360-degree video coding and delivery solutions to enable high quality interactive, on-demand, and live video streaming. The project includes several research thrusts to enable novel joint coding-and-delivery solutions for high quality and robust 360-degree video streaming. For interactive streaming, novel Field-of-View (FoV) adaptive coding structure will be designed to achieve low encoding and decoding latency. Realtime joint optimization of streaming rate adaption and video coding bits allocation based on the predicted FoV will be studied to maximize the rendered video quality. For on-demand streaming, a two-tier video coding and delivery framework will be developed, and the rate allocation and video chunk scheduling between the two tiers will be investigated to strike the desired balance between the rendered video quality and streaming robustness. To facilitate predictive coding and delivery, the project will develop effective algorithms for predicting user FoVs, based on the past FoV trajectory and the audio and visual content through deep learning architectures. Personalized FoV prediction based on other users' view trajectories will also be explored under the framework of recommender systems. Fully-functional 360 video streaming prototypes will be developed and tested in controlled and real network environments to validate and improve the new designs. If successful, the research will lead to new theory and designs for 360-degree video coding and help enable the wide-spread deployment of high-quality and robust 360-video streaming systems. The research findings will be made available through publications, talks, open protocols, and open-source codes, allowing a multitude of developers, researchers, and companies to evolve 360-video streaming. The project will also create valuable research opportunities for graduate and undergraduate students, especially women and minority students. Interactions with industry will be facilitated through workshops and several research centers at the New York University.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

虚拟现实（VR）和增强现实（AR）应用预计将成为未来互联网的下一波“杀手级应用”。VR/AR应用促进了生动的沉浸式虚拟和增强现实体验，并在许多领域创造了巨大的新机会，包括教育，商业，医疗保健和娱乐等。与传统的视频流传输相比，360度视频流传输需要更高的网络带宽和更低的分组递送延迟，并且用户体验质量对网络环境和用户观看行为的动态性高度敏感。为了应对这些独特的挑战，该项目将开发新颖的360度视频编码和交付解决方案，以实现高质量的交互式、点播和实时视频流。 该项目包括几个研究重点，以实现新颖的联合编码和交付解决方案，以实现高质量和强大的360度视频流。对于交互式流媒体，将设计新颖的视场（FoV）自适应编码结构以实现低编码和解码延迟。研究基于预测的FoV的流速率自适应和视频编码比特分配的实时联合优化，以最大化渲染的视频质量。 对于按需流，将开发两层视频编码和交付框架，并研究两层之间的速率分配和视频块调度，以在渲染视频质量和流鲁棒性之间取得所需的平衡。为了促进预测编码和交付，该项目将开发有效的算法来预测用户FoV，基于过去的FoV轨迹以及通过深度学习架构的音频和视频内容。基于其他用户的视图轨迹的个性化FoV预测也将在推荐系统的框架下进行探索。全功能360度视频流原型将在受控和真实的网络环境中开发和测试，以验证和改进新设计。如果成功，该研究将为360度视频编码带来新的理论和设计，并有助于广泛部署高质量和强大的360度视频流系统。研究结果将通过出版物，讲座，开放协议和开源代码提供，允许众多开发人员，研究人员和公司发展360度视频流。该项目还将为研究生和本科生，特别是妇女和少数民族学生创造宝贵的研究机会。通过研讨会和纽约大学的几个研究中心，将促进与工业界的互动。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

A Two-Tier System for On-Demand Streaming of 360 Degree Video Over Dynamic Networks

• DOI: 10.1109/jetcas.2019.2898877
• 发表时间: 2019-02
• 期刊: IEEE Journal on Emerging and Selected Topics in Circuits and Systems
• 影响因子: 4.6
• 作者: Liyang Sun;F. Duanmu;Y. Liu;Yao Wang;Y. Ye;Hang Shi;David H. Dai

Live 360 Degree Video Delivery Based on User Collaboration in a Streaming Flock

• DOI: 10.1109/tmm.2022.3149642
• 发表时间: 2023
• 期刊: IEEE Transactions on Multimedia
• 影响因子: 7.3
• 作者: Liyang Sun;Yixiang Mao;Tongyu Zong;Yong Liu;Yao Wang

Realtime mobile bandwidth and handoff predictions in 4G/5G networks

• DOI: 10.1016/j.comnet.2021.108736
• 发表时间: 2022-01-13
• 期刊: COMPUTER NETWORKS
• 影响因子: 5.6
• 作者: Mei, Lifan;Gou, Jinrui;Liu, Yong
• 通讯作者: Liu, Yong

Tightrope walking in low-latency live streaming: optimal joint adaptation of video rate and playback speed

低时延直播走钢丝：视频码率与播放速度的优化联合适配

• DOI: 10.1145/3458305.3463382
• 发表时间: 2021
• 期刊: MMSys '21: Proceedings of the 12th ACM Multimedia Systems Conference
• 作者: Sun, Liyang;Zong, Tongyu;Wang, Siquan;Liu, Yong;Wang, Yao
• 通讯作者: Wang, Yao

Learning to Predict on Octree for Scalable Point Cloud Geometry Coding

• DOI: 10.1109/mipr54900.2022.00024
• 发表时间: 2022-08
• 期刊: 2022 IEEE 5th International Conference on Multimedia Information Processing and Retrieval (MIPR)
• 作者: Yixiang Mao;Yueyu Hu;Yao Wang