NeTS: Medium: Object-Centric, View-Adaptive and Progressive Coding and Streaming of Point Cloud Video

在互联网上流式传输的大多数视频是由常规摄像机捕获的平面二维（2D）图像的序列。点云视频（PCV）使用点云帧序列记录动态场景的三维（3D）几何形状和颜色信息，每个点云帧是由3D扫描仪或相机阵列捕获的空间中的离散数据点集。捕获的PCV可以由观看者从任何角度在任何观看距离观看，以获得真正沉浸式的视觉体验。部署PCV将在许多领域带来新的机会，包括教育，商业，医疗保健和娱乐等。同时，通过互联网流式传输PCV需要比传统2D视频更高的带宽和更低的延迟;处理PCV也会在源端和接收端产生高计算负载。该项目解决了PCV的通信和计算挑战，并将有助于通过全球互联网广泛部署高质量和强大的PCV流媒体。该项目正在开发以对象为中心、视图自适应、渐进式和边缘感知的PCV编码和流设计，以在网络和观众动态中提供稳健和高质量的观众体验质量（QoE）。它包括几个研究重点：1）项目团队正在研究基于对象的编码方案，最大限度地探索PCV压缩同一对象内点的空间和时间相干性。正在开发一种分层切片结构，用于表示动态八叉树，以实现流传输期间的速率和视场（FoV）自适应。通过考虑其他观众的FoV和PCV中物体的移动来预测观众的FoV; 2）项目团队正在研究渐进式PCV流媒体，随着播放时间的临近，逐渐细化预测FoV中每个区域的空间分辨率。研究人员正在研究新的混合学习基础PCV流媒体解决方案以及低延迟直播流媒体的联合速率和播放速度适应; 3）项目团队正在设计边缘PCV缓存算法，该算法与基于边缘的PCV后处理无缝配合。他们还在探索基于边缘的多播和跨用户FoV预测的多用户交付的收益; 4）正在开发一个功能齐全的PCV流媒体测试平台，通过将专业舞者的PCV以点播和直播方式流式传输给舞蹈学生来进行现代舞蹈教育实验。该项目将产生更好的工具，用于三维人体运动的审查，也可以使许多其他应用受益，包括运动科学/运动医学，职业和物理治疗，康复工程和媒体制作。正在为研究生和本科生，特别是妇女和少数民族学生创造宝贵的研究机会。该项目还为舞蹈学生和从业者创造了参与STEM研究的机会。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Most videos streamed on the Internet are sequences of flat two-dimensional (2D) images captured by regular video cameras. A Point-Cloud Video (PCV) records the three-dimensional (3D) geometry and color information of a dynamic scene using a sequence of point-cloud frames, each of which is a discrete set of data points in space captured by a 3D scanner or a camera array. A captured PCV can be viewed by a viewer from any angle at any viewing distance to obtain a truly immersive visual experience. Deployed PCV will enable new opportunities in many domains, including education, business, healthcare and entertainment, etc. Meanwhile, streaming PCV over the Internet requires significantly higher bandwidth and lower latency than the traditional 2D video; processing PCV also incurs high computation loads on the source and receiver sides. The project addresses the communication and computation challenges of PCV, and will contribute towards the wide deployment of high quality and robust PCV streaming through the global Internet. The project is developing object-centric, view-adaptive, progressive, and edge-aware PCV coding and streaming designs to deliver robust and high-quality viewer Quality-of-Experience (QoE) in the faces of network and viewer dynamics. It includes several research thrusts: 1) The project team is investigating object-based coding schemes that maximally explore the spatial and temporal coherences of points within the same object for PCV compression. A hierarchical slicing structure is being developed for representing dynamic octrees to enable rate and Field-of-View (FoV) adaptations during streaming. A viewer's FoV is predicted by considering the other viewers' FoVs and the movements of objects in a PCV; 2) The project team is studying progressive PCV streaming that gradually refines the spatial resolution of each region in the predicted FoV as its playback time approaches. The researchers are investigating novel hybrid-learning bases PCV streaming solutions and joint rate and playback speed adaptation for low-latency live streaming; 3) The project team is designing edge PCV caching algorithms that work seamlessly with edge-based PCV post-processing. They are also exploring the gains of multi-user delivery from edge-based multicast and cross-user FoV predictions; 4) A fully-functional PCV streaming testbed is being developed to conduct modern dance education experiments by streaming PCVs of professional dancers to dance students in on-demand and live fashions. The project will generate better tools for the review of human motion in three dimensions that can also benefit many other applications, including sports science/sports medicine, occupational and physical therapy, rehabilitation engineering, and media production. Valuable research opportunities are being created for graduate and undergraduate students, especially women and minority students. The project is also creating opportunities for dance students and practitioners to participate in STEM Research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.