Hi-Fit: High fidelity telepresence over best effort networks

高清晰度网真(HDTP)包括使用高清视频捕获场景，通常是办公室、教室、会议室等实际物理位置的用户；即，以每秒30帧的逐行扫描的1920×1080像素的帧大小，通常称为1080p30，并将该视频实况传输给位于不同地理位置的其他用户。HDTP在在线状态、感知、逼真和细节方面实现了高保真交互，通过详细的面部表情、肢体语言、视觉清晰度和保真度提供丰富的用户体验。在目前的全高清分辨率下，HDTP非常逼真，因为它显示了远程位置的实际高分辨率视频，到2020年，由于部署了超高清晰度4K和8K分辨率，它将几乎无法与现实区分开来。由于这种高保真度，HDTP用户不仅可以看到远程用户的脸和外表，甚至可以看到远程用户的眼睛抽搐、磨牙、微妙的身体动作以及通过面部表情和肢体语言表现出来的情感状态等最微小的细节。然而，尽管HDTP系统最近取得了进步，但仍有很多事情要把它们带到大众消费者手中。启用这类系统所需的硬件和软件要求相当苛刻。例如，直到最近，上述大多数HDTP产品都需要购买服务质量有保证的专用网络，以支持用户之间的全高清视频流，这导致了高昂的经常性费用。在此基础上添加3D和/或多视图将使其变得更加笨重和昂贵，以至于在技术上变得不切实际。因此，让这些技术在尽力而为的网络上工作，以显著降低成本并提高实用性仍然是一项挑战。在本研究中，我们设计并实施了一个可用于尽力而为网络上的高保真网真的平台，解决了高清视频、3D视频和多视点视频的处理和流传输方面的许多相关挑战，并将它们集成到高清3D/多视点网真中。

High Definition Telepresence (HDTP) involves capturing a scene, typically users at an actual physical location such as an office, classroom, meeting room, etc., with HD video; i.e., 1920×1080 pixels frame size at 30 frames per second progressive scan commonly referred to as 1080P30, and live streaming that video to other users who are at different geographical locations. HDTP achieves high fidelity interaction in terms of presence, awareness, realism, and details, providing a rich user experience through detailed facial expressions, body language, visual clarity, and fidelity. At its current Full HD resolution, HDTP is quite realistic as it displays an actual high resolution video of the remote location, and by the year 2020 it will become almost indistinguishable from reality due to the deployment of the Ultra High Definition 4K and 8K resolutions. Due to this high fidelity, an HDTP user can see not only the remote users' faces and appearances, but also even the smallest details such as a remote user's eye twitching, teeth grinding, subtle physical movements, and emotional state evident through facial expressions and body language. However, despite recent advancements in HDTP systems, much remains to be bring them to mass consumers. The hardware and software requirements needed to enable such systems are quite demanding. As an example, until recently, most of the above-mentioned HDTP products required purchasing a dedicated network with guaranteed quality of service in order to support Full HD video streaming between users, causing high and recurring expenses. Adding 3D and/or multiview to this would make it even bulkier and more expensive, to the point of becoming technically impractical. Therefore, making these technologies work over best-effort networks in order to significantly reduce cost and increase practicality is still a challenge. In this research, we design and implement a platform that can be used for high fidelity telepresence over best-effort networks by tackling a number of related challenges for the processing and streaming of HD video, 3D video, and multiview video, and integrate them for HD 3D/multiview telepresence.