MetaStream: Live Volumetric Content Capture, Creation, Delivery, and Rendering in Real Time
MetaStream: Live Volumetric Content Capture, Creation, Delivery, and Rendering in Real Time
复制标题
DOI：
10.1145/3570361.3592530
复制
发表时间：
2023-10
期刊：
影响因子：
--
通讯作者：
Yongjie Guan;Xueyu Hou;Nan Wu;Bo Han;Tao Han
中科院分区：
文献类型：
--
作者：
Yongjie Guan;Xueyu Hou;Nan Wu;Bo Han;Tao Han
虽然最近的工作探索了按需流式传输容量内容，但在直播容量视频流上几乎没有付出努力，而直播容量视频流具有比其按需对应物带来更令人兴奋的应用的潜力。为了填补这一关键空白，在本文中，我们提出了MetaStream，据我们所知，这是第一个实用的实时体积内容捕获，创建，交付和渲染系统，用于沉浸式应用程序，如虚拟，增强和混合现实。为了解决处理和流式传输大量3D数据的严格延迟要求的关键挑战，MetaStream将多项创新集成到一个整体系统中，包括动态相机校准，边缘辅助对象分割，跨相机冗余点删除和中心凹体积内容渲染。我们实现了一个原型的MetaStream使用商品设备，并广泛评估其性能。我们的研究结果表明，MetaStream在WiFi网络上以接近每秒30帧的速度实现了低延迟的实时体积视频流。与最先进的系统相比，MetaStream将端到端延迟降低了31.7%，同时将视觉质量提高了12.5%。
While recent work explored streaming volumetric content on-demand, there is little effort on live volumetric video streaming that bears the potential of bringing more exciting applications than its on-demand counterpart. To fill this critical gap, in this paper, we propose MetaStream, which is, to the best of our knowledge, the first practical live volumetric content capture, creation, delivery, and rendering system for immersive applications such as virtual, augmented, and mixed reality. To address the key challenge of the stringent latency requirement for processing and streaming a huge amount of 3D data, MetaStream integrates several innovations into a holistic system, including dynamic camera calibration, edge-assisted object segmentation, cross-camera redundant point removal, and foveated volumetric content rendering. We implement a prototype of MetaStream using commodity devices and extensively evaluate its performance. Our results demonstrate that MetaStream achieves low-latency live volumetric video streaming at close to 30 frames per second on WiFi networks. Compared to state-of-the-art systems, MetaStream reduces end-to-end latency by up to 31.7% while improving visual quality by up to 12.5%.