CNS Core: Medium: Collaborative Research: Scalable Dissemination and Navigation of Video 360 Content for Personalized Viewing

CNS 核心：媒介：协作研究：视频 360 内容的可扩展传播和导航以实现个性化观看

• 批准号: 1901137
• 负责人: Michael Zink
• 金额: $ 79.88万
• 依托单位: University of Massachusetts Amherst
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1901137&HistoricalAwards=false
• 关键词: CNS; Core; Medium; Collaborative; Research

360 video is a form of virtual reality (VR) that allows the viewer to experience media content in an immersive fashion. In contrast to traditional video, 360 video is recorded with a special camera that captures the complete surroundings from almost all directions. Viewers consuming such a video can select the direction they are looking at by using a pointing device on a regular display or through head movement using a head-mounted device. This new format allows a viewer to change their viewing direction when watching the video, e.g., a viewer can watch a sporting event from multiple perspectives on the field. However, creating, storing, and disseminating 360 videos at a large scale over the Internet poses significant challenges. These challenges are the focus of this project, which will develop a new system and framework, called mi360World, to enable smooth delivery of and interaction with 360 video by any user on the Internet. This project, if successful, will significantly improve 360 video delivery, and will enable new and much richer educational, training and entertainment experiences. It will also help train a new class of multimedia systems researchers and practitioners. The mechanisms for delivering a high-quality, personalized 360 video over the Internet to a globally distributed set of users is an unsolved scientific problem that entails the following challenges: 1) Ultra-high Bandwidth; 2) Ultra-low Delay; 3) View Adaptation (to user head movement); 4) Complex video metadata and delivery; 5) Video Quality of Experience (QoE). Traditional video QoE has seen extensive research over the years; however, what contributes to 360 video QoE is much less understood and will require conceiving of and measuring new metrics. The proposed mi360World system incorporates three major research thrusts to address the above challenges: A video creation thrust that enables personalized viewing by generating navigation graphs and cinematographic rules, while maintaining a high QoE and reducing cybersickness. The construction of navigation graphs and inclusion of cinematographic rules represent the main innovations of this project, and are encapsulated in a three-layered metadata representation of the 360 video: a transport layer, a semantic layer, and an interactive story-telling layer. The second thrust focuses on scalable distribution of 360 videos to a global set of diverse viewers, utilizing navigation graphs and cinematographic rules for highly efficient transition-predictive prefetching and caching. The third thrust focuses on QoE and has the goal of devising novel QoE metrics and evaluation methods to assess cybersickness. System architectures and algorithms will be extensively evaluated through simulation, emulation, and benchmarking using testbeds to assess the success of the proposed research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

360视频是虚拟现实(VR)的一种形式，允许观众以身临其境的方式体验媒体内容。与传统视频不同的是，360视频是用一台特殊的摄像头录制的，几乎可以从各个方向捕捉到完整的周围环境。观看这类视频的观众可以通过在常规显示器上使用定点设备或使用头盔设备通过头部移动来选择他们正在观看的方向。这种新格式允许观众在观看视频时改变他们的观看方向，例如，观众可以从球场上的多个视角观看体育赛事。然而，在互联网上大规模创建、存储和传播360个视频会带来巨大的挑战。这些挑战是该项目的重点，该项目将开发一种名为mi360World的新系统和框架，以使互联网上的任何用户都能顺利交付360视频并与其互动。该项目如果成功，将显著改善360视频交付，并将提供新的、更丰富的教育、培训和娱乐体验。它还将有助于培训一批新的多媒体系统研究人员和实践者。通过互联网向全球分布的用户提供高质量、个性化的360视频的机制是一个尚未解决的科学问题，它包括以下挑战：1)超高带宽；2)超低延迟；3)视图自适应(根据用户头部的移动)；4)复杂的视频元数据和传送；5)视频体验质量(QOE)。多年来，传统的视频QOE已经得到了广泛的研究；然而，对360视频QOE的影响却知之甚少，需要构思和衡量新的指标。拟议的mi360World系统包含三个主要研究推力，以应对上述挑战：视频创建推力，通过生成导航图和电影拍摄规则实现个性化观看，同时保持高QOE并减少晕屏。导航图的构建和电影规则的包含代表了该项目的主要创新，并封装在360视频的三层元数据表示中：传输层、语义层和交互式故事层。第二个重点是向全球不同观众组提供360个视频的可扩展分发，利用导航图和电影规则实现高效的过渡预测预取和缓存。第三个重点是QOE，目标是设计新的QOE指标和评估方法来评估晕车。系统架构和算法将通过使用试验台的模拟、仿真和基准测试来广泛评估，以评估拟议研究的成功。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

FOCAS: Practical Video Super Resolution using Foveated Rendering

• DOI: 10.1145/3474085.3475673
• 发表时间: 2021-10
• 期刊: Proceedings of the 29th ACM International Conference on Multimedia
• 作者: Lingdong Wang;M. Hajiesmaili;R. Sitaraman

VOXEL: cross-layer optimization for video streaming with imperfect transmission

• DOI: 10.1145/3485983.3494864
• 发表时间: 2021-12
• 期刊: Proceedings of the 17th International Conference on emerging Networking EXperiments and Technologies
• 作者: Mirko Palmer;Malte Appel;Kevin Spiteri;B. Chandrasekaran;A. Feldmann;R. Sitaraman

Grad: Learning for Overhead-aware Adaptive Video Streaming with Scalable Video Coding

• DOI: 10.1145/3394171.3413512
• 发表时间: 2020-10
• 期刊: Proceedings of the 28th ACM International Conference on Multimedia
• 作者: Yunzhuo Liu;Bo Jiang;Tian Guo;R. Sitaraman;D. Towsley;Xinbing Wang

Semantic-Aware View Prediction for 360-Degree Videos at the 5G Edge

• DOI: 10.1109/ism55400.2022.00025
• 发表时间: 2022-12
• 期刊: 2022 IEEE International Symposium on Multimedia (ISM)
• 作者: Shivi Vats;Jounsup Park;K. Nahrstedt;M. Zink;R. Sitaraman;H. Hellwagner

Full UHD 360-Degree Video Dataset and Modeling of Rate-Distortion Characteristics and Head Movement Navigation

• DOI: 10.1145/3458305.3478447
• 发表时间: 2021-06
• 期刊: Proceedings of the 12th ACM Multimedia Systems Conference
• 作者: Jacob Chakareski;Ridvan Aksu;Viswanathan Swaminathan;M. Zink