CIF: Small: Mobile Immersive Communication: View Sampling and Rate-Distortion Limits

自由视点视频是一种新兴的视觉通信技术，它通过允许用户在任意视点之间动态切换来创建远程场景中的3D沉浸感。它有可能通过实现虚拟人类交通来推动社会进步，并促进全球经济和生活质量。目前，自由视点视频仅限于高端计算环境和演播室类型的设置，由于其更高的带宽和复杂性扩展比单视图视频。此外，视点采样（摄像机位置）和资源分配的基本问题在很大程度上是没有答案的，由于nascency的技术，并使用次优的启发式方法来解决，从而惩罚系统效率。这两个特性将使得自由视点视频不切实际，并且由于其受限的带宽、电池功率和CPU能力而妨碍其更广泛的部署，特别是在移动的设备上。然而，后者已成为随时随地满足计算和通信需求的主要平台，这一趋势在未来只会加速。因此，预期只有通过实现无处不在且无缝的移动的自由视点视频，才可以实现沉浸式通信的全部潜力。该项目旨在通过在信号表示，无线视频通信和用户行为建模，将整体集成的协调一致的进步来实现这一目标。 这项调查所带来的进步将对涉及通过多个视角进行实时视频通信的各个领域产生广泛影响，包括远程医疗，远程呈现和远程会诊，远程监控和控制，娱乐（3D和自由视点电视），游戏和虚拟世界，以人为本的传感和连接社区应用。该项目将追求以下技术目标：（i）描述沉浸式移动的通信中视点空间采样、速率分配和信号保真度之间的基本权衡;（ii）在上行链路通信中，在视点和数据单元级别，导出最优采样策略;（iii）在下行链路通信中，设计用于多视图广播的新颖视图和速率可缩放编码，并导出作为广播速率的函数的最佳视图嵌入策略;（iv）表征用于移动的客户端之间的本地自组织通信的最优调度策略;以及（v）整合能量节约和表征电池寿命与多视图应用性能之间的权衡。

Free-viewpoint video is an emerging technology for visual communication that creates the sensation of 3D immersion in the remote scene by allowing the user to dynamically switch between arbitrary viewpoints. It has the potential to advance society by enabling virtual human transportation and boost the global economy and quality of life. At present, free-viewpoint video is limited to high-end computing environments and studio-type settings, due to its higher bandwidth and complexity expansion over single-view video. Furthermore, the fundamental questions of viewpoint sampling (camera location) and resource allocation across the captured views are largely unanswered, due to the nascency of the technology, and are addressed using suboptimal heuristic approaches, thus penalizing system efficiency. These two characteristics would otherwise make free-viewpoint video impractical and preclude its broader deployment, in particular on mobile devices, due to their constrained bandwidth, battery power, and CPU capabilities. However, the latter have become a primary platform for computing and communication needs, anywhere and anytime, a trend that will only accelerate in the future. Thus, it is anticipated that only by enabling ubiquitous and seamless mobile free-viewpoint video may the full potential of immersive communication be achieved. This project seeks to achieve this goal via concerted advances in signal representation, wireless video communication, and user-action modeling that will be integrated holistically. The advances delivered by this investigation will have broad impact across diverse fields that involve live video communication via multiple viewpoints, including telemedicine, telepresence and telecollaboration, remote monitoring and control, entertainment (3D and free-viewpoint TV), gaming and virtual worlds, people-centric sensing and connected-community applications.The project will pursue the following technical thrusts: (i) Characterization of the fundamental trade-offs between viewpoint space sampling, rate allocation, and signal fidelity in immersive mobile communication; (ii) Derivation of the optimal sampling policy, at the view and data-unit levels, in uplink communication; (iii) Design of novel view and rate scalable coding for multi-view broadcast and derivation of the optimal view embedding policy, as a function of the broadcast rate, in downlink communication; (iv) Characterization of the optimal scheduling policy for local ad-hoc communication between mobile clients; and (v) Integration of energy conservation and characterization of the trade-offs between battery lifetime and multi-view application performance.