NeTS: Medium: Object-Centric, View-Adaptive and Progressive Coding and Streaming of Point Cloud Video

大多数在互联网上流传的视频都是由普通摄像机拍摄的平面二维(2D)图像序列。点云视频(PCV)使用一系列点云帧记录动态场景的三维(3D)几何和颜色信息，每个点云帧是由3D扫描仪或相机阵列捕获的空间中的一组离散数据点。捕捉到的PCV可以由观众从任何角度、任何观看距离观看，以获得真正身临其境的视觉体验。部署PCV将在教育、商业、医疗和娱乐等多个领域带来新的机遇。同时，互联网上的PCV流媒体比传统的2D视频需要更高的带宽和更低的延迟；处理PCV也会在源端和接收端产生高计算负载。该项目解决了PCV的通信和计算挑战，并将有助于通过全球互联网广泛部署高质量和健壮的PCV流。该项目正在开发以对象为中心、视图自适应、渐进式和边缘感知的PCV编码和流设计，以在网络和观众动态面前提供强大而高质量的观众体验质量(QOE)。它包括几个研究主题：1)项目团队正在研究基于对象的编码方案，该方案最大限度地探索用于PCV压缩的同一对象内的点的空间和时间一致性。正在开发用于表示动态八叉树的分层切片结构，以实现流传输期间的速率和视场(FOV)自适应。观众的视场是通过考虑其他观众的视场和PCV中对象的移动来预测的；2)项目团队正在研究渐进的PCV流，该流随着预测视场中每个区域的回放时间的临近而逐渐细化其空间分辨率。研究人员正在研究新的混合学习基础PCV流媒体解决方案，以及用于低延迟直播的联合速率和回放速度适配；3)项目组正在设计与基于边缘的PCV后处理无缝协作的边缘PCV缓存算法。他们还在探索从基于边缘的组播和跨用户FoV预测中获得的多用户交付的收益；4)正在开发一种全功能的PCV流媒体试验台，通过以点播和直播的方式向舞蹈学生播放专业舞者的PCV，来进行现代舞蹈教育实验。该项目将产生更好的工具，用于从三个维度审查人体运动，这也可以使许多其他应用受益，包括运动科学/运动医学、职业和物理治疗、康复工程和媒体制作。正在为研究生和本科生，特别是女性和少数族裔学生创造宝贵的研究机会。该项目还为舞蹈学生和实践者创造了参与STEM研究的机会。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Most videos streamed on the Internet are sequences of flat two-dimensional (2D) images captured by regular video cameras. A Point-Cloud Video (PCV) records the three-dimensional (3D) geometry and color information of a dynamic scene using a sequence of point-cloud frames, each of which is a discrete set of data points in space captured by a 3D scanner or a camera array. A captured PCV can be viewed by a viewer from any angle at any viewing distance to obtain a truly immersive visual experience. Deployed PCV will enable new opportunities in many domains, including education, business, healthcare and entertainment, etc. Meanwhile, streaming PCV over the Internet requires significantly higher bandwidth and lower latency than the traditional 2D video; processing PCV also incurs high computation loads on the source and receiver sides. The project addresses the communication and computation challenges of PCV, and will contribute towards the wide deployment of high quality and robust PCV streaming through the global Internet. The project is developing object-centric, view-adaptive, progressive, and edge-aware PCV coding and streaming designs to deliver robust and high-quality viewer Quality-of-Experience (QoE) in the faces of network and viewer dynamics. It includes several research thrusts: 1) The project team is investigating object-based coding schemes that maximally explore the spatial and temporal coherences of points within the same object for PCV compression. A hierarchical slicing structure is being developed for representing dynamic octrees to enable rate and Field-of-View (FoV) adaptations during streaming. A viewer's FoV is predicted by considering the other viewers' FoVs and the movements of objects in a PCV; 2) The project team is studying progressive PCV streaming that gradually refines the spatial resolution of each region in the predicted FoV as its playback time approaches. The researchers are investigating novel hybrid-learning bases PCV streaming solutions and joint rate and playback speed adaptation for low-latency live streaming; 3) The project team is designing edge PCV caching algorithms that work seamlessly with edge-based PCV post-processing. They are also exploring the gains of multi-user delivery from edge-based multicast and cross-user FoV predictions; 4) A fully-functional PCV streaming testbed is being developed to conduct modern dance education experiments by streaming PCVs of professional dancers to dance students in on-demand and live fashions. The project will generate better tools for the review of human motion in three dimensions that can also benefit many other applications, including sports science/sports medicine, occupational and physical therapy, rehabilitation engineering, and media production. Valuable research opportunities are being created for graduate and undergraduate students, especially women and minority students. The project is also creating opportunities for dance students and practitioners to participate in STEM Research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.