CIF: Small: Mobile Immersive Communication: View Sampling and Rate-Distortion Limits

自由视点视频是一种新兴的视觉交流技术，它允许用户在任意视点之间动态切换，从而在远程场景中创造出3D沉浸感。它有可能通过实现虚拟人类交通来推动社会发展，并促进全球经济和生活质量。目前，自由视点视频由于其比单视点视频具有更高的带宽和复杂性扩展，仅限于高端计算环境和工作室类型的设置。此外，由于该技术尚不成熟，视点采样（摄像机位置）和捕获视图之间的资源分配的基本问题在很大程度上没有答案，并且使用次优启发式方法来解决，从而降低了系统效率。这两个特点将使自由视点视频变得不切实际，并阻碍其更广泛的部署，特别是在移动设备上，因为它们的带宽、电池电量和CPU能力受到限制。然而，后者已经成为随时随地计算和通信需求的主要平台，这一趋势在未来只会加速。因此，预计只有通过实现无处不在和无缝的移动自由视点视频，才能实现沉浸式通信的全部潜力。该项目旨在通过信号表示、无线视频通信和用户操作建模的协同进步来实现这一目标，这些将被整体集成。这项调查所带来的进步将在多个领域产生广泛的影响，包括通过多视点进行实时视频通信，包括远程医疗、远程呈现和远程协作、远程监控和控制、娱乐（3D和自由视点电视）、游戏和虚拟世界、以人为中心的传感和连接社区应用。该项目将追求以下技术重点：(i)描述沉浸式移动通信中视点空间采样、速率分配和信号保真度之间的基本权衡；（ii）在上行通信的视图和数据单元级别推导出最优采样策略；（iii）为多视点广播设计新颖的视点和速率可伸缩编码，并推导出下行通信中作为广播速率函数的最优视点嵌入策略；（iv）移动客户端间本地自组织通信的最优调度策略表征；(v)集成节能和表征电池寿命和多视图应用性能之间的权衡。

Free-viewpoint video is an emerging technology for visual communication that creates the sensation of 3D immersion in the remote scene by allowing the user to dynamically switch between arbitrary viewpoints. It has the potential to advance society by enabling virtual human transportation and boost the global economy and quality of life. At present, free-viewpoint video is limited to high-end computing environments and studio-type settings, due to its higher bandwidth and complexity expansion over single-view video. Furthermore, the fundamental questions of viewpoint sampling (camera location) and resource allocation across the captured views are largely unanswered, due to the nascency of the technology, and are addressed using suboptimal heuristic approaches, thus penalizing system efficiency. These two characteristics would otherwise make free-viewpoint video impractical and preclude its broader deployment, in particular on mobile devices, due to their constrained bandwidth, battery power, and CPU capabilities. However, the latter have become a primary platform for computing and communication needs, anywhere and anytime, a trend that will only accelerate in the future. Thus, it is anticipated that only by enabling ubiquitous and seamless mobile free-viewpoint video may the full potential of immersive communication be achieved. This project seeks to achieve this goal via concerted advances in signal representation, wireless video communication, and user-action modeling that will be integrated holistically. The advances delivered by this investigation will have broad impact across diverse fields that involve live video communication via multiple viewpoints, including telemedicine, telepresence and telecollaboration, remote monitoring and control, entertainment (3D and free-viewpoint TV), gaming and virtual worlds, people-centric sensing and connected-community applications.The project will pursue the following technical thrusts: (i) Characterization of the fundamental trade-offs between viewpoint space sampling, rate allocation, and signal fidelity in immersive mobile communication; (ii) Derivation of the optimal sampling policy, at the view and data-unit levels, in uplink communication; (iii) Design of novel view and rate scalable coding for multi-view broadcast and derivation of the optimal view embedding policy, as a function of the broadcast rate, in downlink communication; (iv) Characterization of the optimal scheduling policy for local ad-hoc communication between mobile clients; and (v) Integration of energy conservation and characterization of the trade-offs between battery lifetime and multi-view application performance.