NeTS: Small: Dynamic Predictive Streaming of 360 Degree Video

虚拟现实（VR）和增强现实（AR）应用预计将成为未来互联网的下一波“杀手级应用”。 VR/AR应用促进了生动的沉浸式虚拟和增强现实体验，并在许多领域创造了巨大的新机遇，包括教育、商业、医疗保健和娱乐等。许多VR/AR应用涉及360度视频场景流。与传统视频流相比，360度视频流需要更高的网络带宽和更低的数据包传输延迟，并且用户的体验质量对网络环境和用户观看行为的动态高度敏感。为了应对这些独特的挑战，该项目将开发新颖的 360 度视频编码和交付解决方案，以实现高质量的交互式、点播和实时视频流。 该项目包括多项研究重点，旨在实现新颖的联合编码和交付解决方案，以实现高质量和强大的 360 度视频流。对于交互式流媒体，将设计新颖的视场（FoV）自适应编码结构以实现低编码和解码延迟。将研究基于预测视场的流速率自适应和视频编码比特分配的实时联合优化，以最大化渲染视频质量。 对于点播流媒体，将开发两层视频编码和交付框架，并将研究两层之间的速率分配和视频块调度，以在渲染视频质量和流媒体鲁棒性之间取得所需的平衡。为了促进预测编码和交付，该项目将基于过去的 FoV 轨迹以及通过深度学习架构的音频和视频内容，开发有效的算法来预测用户 FoV。基于其他用户观看轨迹的个性化视场预测也将在推荐系统的框架下进行探索。全功能的 360 度视频流原型将在受控的真实网络环境中进行开发和测试，以验证和改进新设计。如果成功，该研究将带来 360 度视频编码的新理论和设计，并有助于实现高质量和强大的 360 度视频流系统的广泛部署。研究结果将通过出版物、演讲、开放协议和开源代码的形式公布，让众多开发人员、研究人员和公司能够发展 360 度视频流。该项目还将为研究生和本科生，特别是女性和少数民族学生创造宝贵的研究机会。纽约大学的研讨会和多个研究中心将促进与行业的互动。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Virtual Reality (VR) and Augmented Reality (AR) applications are projected to be the next wave of "Killer Apps" in the future Internet. VR/AR applications facilitate vivid immersive virtual and augmented reality experience and create tremendous new opportunities in many domains, including education, business, healthcare, and entertainment, etc. Many VR/AR applications involve streaming of 360-degree video scenes. Compared with the traditional video streaming, 360-degree video streaming requires much higher network bandwidth and much lower packet delivery latency, and user's quality of experience is highly sensitive to the dynamics in both network environment and user viewing behaviors. Addressing these unique challenges, this project will develop novel 360-degree video coding and delivery solutions to enable high quality interactive, on-demand, and live video streaming. The project includes several research thrusts to enable novel joint coding-and-delivery solutions for high quality and robust 360-degree video streaming. For interactive streaming, novel Field-of-View (FoV) adaptive coding structure will be designed to achieve low encoding and decoding latency. Realtime joint optimization of streaming rate adaption and video coding bits allocation based on the predicted FoV will be studied to maximize the rendered video quality. For on-demand streaming, a two-tier video coding and delivery framework will be developed, and the rate allocation and video chunk scheduling between the two tiers will be investigated to strike the desired balance between the rendered video quality and streaming robustness. To facilitate predictive coding and delivery, the project will develop effective algorithms for predicting user FoVs, based on the past FoV trajectory and the audio and visual content through deep learning architectures. Personalized FoV prediction based on other users' view trajectories will also be explored under the framework of recommender systems. Fully-functional 360 video streaming prototypes will be developed and tested in controlled and real network environments to validate and improve the new designs. If successful, the research will lead to new theory and designs for 360-degree video coding and help enable the wide-spread deployment of high-quality and robust 360-video streaming systems. The research findings will be made available through publications, talks, open protocols, and open-source codes, allowing a multitude of developers, researchers, and companies to evolve 360-video streaming. The project will also create valuable research opportunities for graduate and undergraduate students, especially women and minority students. Interactions with industry will be facilitated through workshops and several research centers at the New York University.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.