CIF: Small: Mobile Immersive Communication: View Sampling and Rate-Distortion Limits

自由视点视频是一种新兴的视觉交流技术，通过允许用户在任意视点之间动态切换，在远程场景中创建3D沉浸感。它有可能通过实现虚拟人交通来推动社会进步，并提高全球经济和生活质量。目前，自由视点视频仅限于高端计算环境和演播室类型的环境，与单视视频相比，其带宽和复杂性扩展更高。此外，由于该技术的新颖性，视点采样(摄像机位置)和跨捕获视点的资源分配的基本问题在很大程度上没有得到回答，并且使用次优的启发式方法来解决，从而损害了系统的效率。这两个特性将使自由视点视频变得不切实际，并阻止其更广泛的部署，特别是在移动设备上，因为它们的带宽、电池功率和CPU功能有限。然而，后者已经成为随时随地满足计算和通信需求的主要平台，这一趋势在未来只会加速。因此，预计只有实现无处不在的无缝移动自由视点视频，才能实现沉浸式通信的全部潜力。该项目寻求通过在信号表示、无线视频通信和将被整体集成的用户动作建模方面的协调发展来实现这一目标。这项研究取得的进展将对不同领域产生广泛影响，这些领域涉及通过多个视角进行现场视频通信，包括远程医疗、远程呈现和远程协作、远程监测和控制、娱乐(3D和自由视点电视)、游戏和虚拟世界、以人为中心的传感和互联社区应用。该项目将追求以下技术进展：(I)描述沉浸式移动通信中视点空间采样、速率分配和信号保真度之间的基本权衡；(Ii)在上行链路通信中，在视点和数据单元级别推导最佳采样策略；(Iii)设计用于多视点广播的新颖视点和速率可分级编码，并作为下行链路通信中广播速率的函数导出最优视点嵌入策略；(Iv)表征用于移动客户端之间的本地自组织通信的最佳调度策略；以及(V)整合能量守恒和表征电池寿命和多视点应用性能之间的权衡。

Free-viewpoint video is an emerging technology for visual communication that creates the sensation of 3D immersion in the remote scene by allowing the user to dynamically switch between arbitrary viewpoints. It has the potential to advance society by enabling virtual human transportation and boost the global economy and quality of life. At present, free-viewpoint video is limited to high-end computing environments and studio-type settings, due to its higher bandwidth and complexity expansion over single-view video. Furthermore, the fundamental questions of viewpoint sampling (camera location) and resource allocation across the captured views are largely unanswered, due to the nascency of the technology, and are addressed using suboptimal heuristic approaches, thus penalizing system efficiency. These two characteristics would otherwise make free-viewpoint video impractical and preclude its broader deployment, in particular on mobile devices, due to their constrained bandwidth, battery power, and CPU capabilities. However, the latter have become a primary platform for computing and communication needs, anywhere and anytime, a trend that will only accelerate in the future. Thus, it is anticipated that only by enabling ubiquitous and seamless mobile free-viewpoint video may the full potential of immersive communication be achieved. This project seeks to achieve this goal via concerted advances in signal representation, wireless video communication, and user-action modeling that will be integrated holistically. The advances delivered by this investigation will have broad impact across diverse fields that involve live video communication via multiple viewpoints, including telemedicine, telepresence and telecollaboration, remote monitoring and control, entertainment (3D and free-viewpoint TV), gaming and virtual worlds, people-centric sensing and connected-community applications.The project will pursue the following technical thrusts: (i) Characterization of the fundamental trade-offs between viewpoint space sampling, rate allocation, and signal fidelity in immersive mobile communication; (ii) Derivation of the optimal sampling policy, at the view and data-unit levels, in uplink communication; (iii) Design of novel view and rate scalable coding for multi-view broadcast and derivation of the optimal view embedding policy, as a function of the broadcast rate, in downlink communication; (iv) Characterization of the optimal scheduling policy for local ad-hoc communication between mobile clients; and (v) Integration of energy conservation and characterization of the trade-offs between battery lifetime and multi-view application performance.