Collaborative Research: CNS Core: Medium: Panoptes: Next Generation Multi-Perspective Video Delivery at Internet Scale

合作研究：CNS 核心：媒介：Panoptes：互联网规模的下一代多视角视频传输

• 批准号: 1956190
• 负责人: Ramesh Govindan
• 金额: $ 60万
• 依托单位: University of Southern California
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1956190&HistoricalAwards=false
• 关键词: Collaborative; Research; CNS; Core; Medium

The success of streaming video has generated interest in newer forms of multi-perspective video content, such as those generated by 360-degree cameras, multi-angle camera arrays, or light-field cameras. The immersive experience provided by these cameras can enhance user satisfaction in domains including sports, training (e.g., construction safety), and virtual exploration (e.g., college walkthroughs, historical sites). With content from these cameras, users do not just passively consume content, but may interactively traverse the content along many different paths from a perspective of their choice, with different users observing different perspectives of the same content. To support this at Internet-scale is challenging; client players must be able to switch perspectives with low latency, the perspectives may need to be generated on demand by video servers, and the infrastructure must support a variety of devices for capture and consumption of this content.This project explores architectural enhancements, algorithms, and techniques to deliver multi-perspective video at Internet-scale. It couples delivery optimization with video coding and human computer interaction. The project will develop interactivity abstractions by which content publishers can specify the range of perspectives users are permitted to choose from at each point in the video. The interactivity specified in a video will (i) drive perspective coding and novel dynamic perspective generation algorithms; (ii) enable infrastructure provisioning to meet Content Delivery Network (CDN) storage and cost constraints; and (iii) will guide adaptive perspective retrieval from CDN servers or from nearby caches. Finally, the project will explore methods to predict and guide user behavior to improve delivery quality based on user studies. By enabling more responsive and predictable user experience for multi-perspective video at Internet scale, and by designing networks that can cope with growth in such video, the project will impact the media, IT, education, and networking industries. It will convene an Industrial Advisory Board comprising relevant practitioners and organize workshops to foster collaborations. The research will not only extensively involve graduate students, but also undergraduate students from underrepresented communities, will create new curriculum material, as well as a new undergraduate design project that involves students ranging from the freshman to senior year with strong participation from women and underrepresented students. The results from the project will be available at: https://engineering.purdue.edu/~isl/multi-perspective-video.htmland https://nsl.usc.edu/projects/multi-perspective-video/The results will be available throughout the duration of the project, and for three years after the project ends.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

流媒体视频的成功引起了人们对新形式的多视角视频内容的兴趣，例如由360度摄像机、多角度摄像机阵列或光场摄像机生成的视频内容。这些摄像头提供的沉浸式体验可以提高用户在体育、培训（如建筑安全）和虚拟探索（如大学演练、历史遗迹）等领域的满意度。通过这些摄像头拍摄的内容，用户不仅可以被动地消费内容，还可以从他们选择的角度，通过许多不同的路径交互地浏览内容，不同的用户可以从不同的角度观察同一内容。在互联网规模上支持这一点是具有挑战性的；客户端播放器必须能够以低延迟切换透视图，透视图可能需要由视频服务器按需生成，并且基础设施必须支持各种设备来捕获和消费此内容。该项目探索了架构增强、算法和技术，以在互联网规模上提供多视角视频。它将传输优化与视频编码和人机交互相结合。该项目将开发交互性抽象，内容发布者可以通过它指定允许用户在视频的每个点上选择的视角范围。在视频中指定的交互性将(i)驱动透视编码和新颖的动态透视生成算法；（ii）使基础设施供应能够满足内容分发网络（CDN）存储和成本限制；（iii）将指导从CDN服务器或附近缓存中进行自适应视角检索。最后，该项目将探索基于用户研究的预测和指导用户行为的方法，以提高交付质量。通过为互联网规模的多视角视频提供更灵敏和可预测的用户体验，以及通过设计能够应对此类视频增长的网络，该项目将影响媒体、IT、教育和网络行业。它将召集一个由相关从业人员组成的工业咨询委员会，并举办讲习班以促进合作。这项研究不仅将广泛涉及研究生，还将涉及来自代表性不足社区的本科生，将创建新的课程材料，以及一个新的本科设计项目，该项目涉及从大一到大四的学生，其中包括女性和代表性不足的学生的强烈参与。该项目的结果将在https://engineering.purdue.edu/~isl/multi-perspective-video.htmland https://nsl.usc.edu/projects/multi-perspective-video/The上提供，结果将在整个项目期间以及项目结束后的三年内提供。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Fractional Motion Estimation for Point Cloud Compression

• DOI: 10.1109/dcc52660.2022.00045
• 发表时间: 2022-02
• 期刊: 2022 Data Compression Conference (DCC)
• 作者: Haoran Hong;Eduardo Pavez;Antonio Ortega;R. Watanabe;Keisuke Nonaka

Cylindrical Coordinates for Lidar Point Cloud Compression

• DOI: 10.1109/icip42928.2021.9506448
• 发表时间: 2021-06
• 期刊: 2021 IEEE International Conference on Image Processing (ICIP)
• 作者: Shashank N. Sridhara;Eduardo Pavez;Antonio Ortega

Motion Estimation And Filtered Prediction For Dynamic Point Cloud Attribute Compression

动态点云属性压缩的运动估计和滤波预测

• DOI: 10.1109/pcs56426.2022.10018071
• 发表时间: 2022
• 期刊: PCS 2022
• 作者: Hong, Haoran;Pavez, Eduardo;Ortega, Antonio;Watanabe, Ryosuke;Nonaka, Keisuke
• 通讯作者: Nonaka, Keisuke

Point Cloud Attribute Compression Via Chroma Subsampling

通过色度子采样进行点云属性压缩

• DOI: 10.1109/icassp43922.2022.9746352
• 发表时间: 2022
• 期刊: Speech and Signal Processing (ICASSP
• 作者: Sridhara, Shashank N.;Pavez, Eduardo;Ortega, Antonio;Watanabe, Ryosuke;Nonaka, Keisuke
• 通讯作者: Nonaka, Keisuke