SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence
SLAMCast: Large-Scale, Real-Time 3D Reconstruction and Streaming for Immersive Multi-Client Live Telepresence
复制标题
DOI：
10.1109/tvcg.2019.2899231
复制
发表时间：
2018-05
影响因子：
5.2
通讯作者：
P. Stotko;S. Krumpen;M. Hullin;Michael Weinmann;R. Klein
中科院分区：
文献类型：
--
作者：
P. Stotko;S. Krumpen;M. Hullin;Michael Weinmann;R. Klein
近年来，RGB-D传感器数据的实时3D场景重建以及VR/AR设置中此类数据的探索都取得了巨大进展。但是，这两个组件与远程呈现系统的结合都面临着重大的技术挑战。到目前为止，所有提出的方法都要求输入和输出设备，计算资源和传输带宽，并且它们没有达到诸如远程协作之类的应用所需的即时性水平。在这里，我们介绍了我们认为是实时捕获和对静态3D场景的多用户探索的第一个实用的客户服务器系统。我们的系统基于这样的观察，即交互式帧速率足以捕获和重建，并且仅在客户端站点上需要实时性能才能在渲染3D模型时实现无滞后视图更新。从这个见解开始，我们通过引入一个新颖的线程gpu哈希地图数据结构来扩展先前的体素块哈希框架，该框架在大规模同步的检索，插入和在线程级别上删除条目。我们进一步提出了一种针对体积数据的新型传输方案，该方案专门针对行进多维数据集的几何重建，并使服务器和勘探客户端之间的带宽减少了90％。最终的系统对网络带宽，延迟和客户端计算提出了非常适度的要求，这使其能够完全依赖于包括移动设备在内的消费级硬件。我们证明，我们的技术在提供任何数量的客户时，即使在网络中断期间也可以达到最先进的表示的准确性。
Real-time 3D scene reconstruction from RGB-D sensor data, as well as the exploration of such data in VR/AR settings, has seen tremendous progress in recent years. The combination of both these components into telepresence systems, however, comes with significant technical challenges. All approaches proposed so far are extremely demanding on input and output devices, compute resources and transmission bandwidth, and they do not reach the level of immediacy required for applications such as remote collaboration. Here, we introduce what we believe is the first practical client-server system for real-time capture and many-user exploration of static 3D scenes. Our system is based on the observation that interactive frame rates are sufficient for capturing and reconstruction, and real-time performance is only required on the client site to achieve lag-free view updates when rendering the 3D model. Starting from this insight, we extend previous voxel block hashing frameworks by introducing a novel thread-safe GPU hash map data structure that is robust under massively concurrent retrieval, insertion and removal of entries on a thread level. We further propose a novel transmission scheme for volume data that is specifically targeted to Marching Cubes geometry reconstruction and enables a 90% reduction in bandwidth between server and exploration clients. The resulting system poses very moderate requirements on network bandwidth, latency and client-side computation, which enables it to rely entirely on consumer-grade hardware, including mobile devices. We demonstrate that our technique achieves state-of-the-art representation accuracy while providing, for any number of clients, an immersive and fluid lag-free viewing experience even during network outages.