NeTS: Medium: Object-Centric, View-Adaptive and Progressive Coding and Streaming of Point Cloud Video

NeTS：Medium：以对象为中心、视图自适应和渐进式的点云视频编码和流式传输

• 批准号: 2312839
• 负责人: Yong Liu
• 金额: $ 120万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2027-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2312839&HistoricalAwards=false
• 关键词: NeTS; Medium; Object; Centric; View

Most videos streamed on the Internet are sequences of flat two-dimensional (2D) images captured by regular video cameras. A Point-Cloud Video (PCV) records the three-dimensional (3D) geometry and color information of a dynamic scene using a sequence of point-cloud frames, each of which is a discrete set of data points in space captured by a 3D scanner or a camera array. A captured PCV can be viewed by a viewer from any angle at any viewing distance to obtain a truly immersive visual experience. Deployed PCV will enable new opportunities in many domains, including education, business, healthcare and entertainment, etc. Meanwhile, streaming PCV over the Internet requires significantly higher bandwidth and lower latency than the traditional 2D video; processing PCV also incurs high computation loads on the source and receiver sides. The project addresses the communication and computation challenges of PCV, and will contribute towards the wide deployment of high quality and robust PCV streaming through the global Internet. The project is developing object-centric, view-adaptive, progressive, and edge-aware PCV coding and streaming designs to deliver robust and high-quality viewer Quality-of-Experience (QoE) in the faces of network and viewer dynamics. It includes several research thrusts: 1) The project team is investigating object-based coding schemes that maximally explore the spatial and temporal coherences of points within the same object for PCV compression. A hierarchical slicing structure is being developed for representing dynamic octrees to enable rate and Field-of-View (FoV) adaptations during streaming. A viewer's FoV is predicted by considering the other viewers' FoVs and the movements of objects in a PCV; 2) The project team is studying progressive PCV streaming that gradually refines the spatial resolution of each region in the predicted FoV as its playback time approaches. The researchers are investigating novel hybrid-learning bases PCV streaming solutions and joint rate and playback speed adaptation for low-latency live streaming; 3) The project team is designing edge PCV caching algorithms that work seamlessly with edge-based PCV post-processing. They are also exploring the gains of multi-user delivery from edge-based multicast and cross-user FoV predictions; 4) A fully-functional PCV streaming testbed is being developed to conduct modern dance education experiments by streaming PCVs of professional dancers to dance students in on-demand and live fashions. The project will generate better tools for the review of human motion in three dimensions that can also benefit many other applications, including sports science/sports medicine, occupational and physical therapy, rehabilitation engineering, and media production. Valuable research opportunities are being created for graduate and undergraduate students, especially women and minority students. The project is also creating opportunities for dance students and practitioners to participate in STEM Research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

互联网上的大多数视频流都是由普通摄像机拍摄的平面二维（2D）图像序列。点云视频（PCV）使用一系列点云帧记录动态场景的三维（3D）几何和颜色信息，每个点云帧都是由3D扫描仪或相机阵列捕获的空间中的离散数据点集。捕获的PCV可以被观看者从任何角度、任何观看距离观看，以获得真正身临其境的视觉体验。部署PCV将在许多领域带来新的机遇，包括教育、商业、医疗保健和娱乐等。同时，与传统的2D视频相比，通过互联网传输PCV需要更高的带宽和更低的延迟；处理PCV也会在源端和接收端产生很高的计算负荷。该项目解决了PCV的通信和计算挑战，并将有助于通过全球互联网广泛部署高质量和强大的PCV流。该项目正在开发以对象为中心、视图自适应、渐进式和边缘感知的PCV编码和流设计，以在面对网络和观众动态时提供强大和高质量的观众体验质量（QoE）。它包括几个研究重点：1)项目团队正在研究基于对象的编码方案，最大限度地探索PCV压缩中同一对象内点的空间和时间相干性。正在开发一种分层切片结构，用于表示动态八叉树，以实现流媒体期间的速率和视场（FoV）适应。通过考虑其他观看者的视场和PCV中物体的运动来预测观看者的视场；2)项目组正在研究渐进式PCV流，随着播放时间的临近，逐步细化预测视场中每个区域的空间分辨率。研究人员正在研究新的混合学习基础的PCV流解决方案，以及低延迟直播的联合速率和播放速度适应；3)项目组正在设计与基于边缘的PCV后处理无缝配合的边缘PCV缓存算法。他们还从基于边缘的多播和跨用户FoV预测中探索多用户传输的收益；4)开发功能齐全的PCV流媒体试验台，将专业舞者的PCV以点播和直播的方式流媒体给舞蹈学生，进行现代舞教育实验。该项目将为三维人体运动研究提供更好的工具，这也有利于许多其他应用，包括运动科学/运动医学、职业和物理治疗、康复工程和媒体制作。为研究生和本科生，特别是女性和少数民族学生创造了宝贵的研究机会。该项目还为舞蹈学生和从业者创造了参与STEM研究的机会。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。