Hi-Fit: High fidelity telepresence over best effort networks

点击翻译按钮获取中文摘要

High Definition Telepresence (HDTP) involves capturing a scene, typically users at an actual physical location such as an office, classroom, meeting room, etc., with HD video; i.e., 1920×1080 pixels frame size at 30 frames per second progressive scan commonly referred to as 1080P30, and live streaming that video to other users who are at different geographical locations. HDTP achieves high fidelity interaction in terms of presence, awareness, realism, and details, providing a rich user experience through detailed facial expressions, body language, visual clarity, and fidelity. At its current Full HD resolution, HDTP is quite realistic as it displays an actual high resolution video of the remote location, and by the year 2020 it will become almost indistinguishable from reality due to the deployment of the Ultra High Definition 4K and 8K resolutions. Due to this high fidelity, an HDTP user can see not only the remote users' faces and appearances, but also even the smallest details such as a remote user's eye twitching, teeth grinding, subtle physical movements, and emotional state evident through facial expressions and body language. However, despite recent advancements in HDTP systems, much remains to be bring them to mass consumers. The hardware and software requirements needed to enable such systems are quite demanding. As an example, until recently, most of the above-mentioned HDTP products required purchasing a dedicated network with guaranteed quality of service in order to support Full HD video streaming between users, causing high and recurring expenses. Adding 3D and/or multiview to this would make it even bulkier and more expensive, to the point of becoming technically impractical. Therefore, making these technologies work over best-effort networks in order to significantly reduce cost and increase practicality is still a challenge. In this research, we design and implement a platform that can be used for high fidelity telepresence over best-effort networks by tackling a number of related challenges for the processing and streaming of HD video, 3D video, and multiview video, and integrate them for HD 3D/multiview telepresence.