Enhanced video for shared reality environments

为了将视频会议技术带入一个新的质量维度，我们正在开发共享现实环境，它支持高带宽，高保真的远程呈现。 该环境包括两个房间，视频投影，摄像机的视频采集，多声道音响系统的空间化音频渲染，和一个振动机械平台，以再现振动刺激在地板水平。与传统的视频会议环境不同，共享现实为用户提供了一种在公共共享空间中的视频和音频沉浸感。 远程用户看起来就像是本地环境的一部分，并且可以在视图中操作合成（计算机生成的）对象。 为了提高共享现实体验的质量，这项研究现在寻求利用输入视频中可用的额外提示，以提供更快，更高分辨率的显示，这些显示不会受到阻挡投影仪输出的用户投射的阴影的影响，并且可以呈现适合当前参与者位置的视图。 所采用的一些技术包括图像拼接、鲁棒跟踪和姿态获取、动态背景分割和立体深度估计。 应用在一起，这项研究有可能支持分布式沉浸式可视化的需求，如工程师团队合作解决复杂问题所需的，这些问题需要彼此之间以及合成模型之间的交互。

In order to bring videoconferencing technology to a new dimension of quality, we are developing the Shared Reality Environment, which supports high-bandwidth, high-fidelity telepresence.  The environment consists of two rooms with video projection, cameras for video acquisition, multichannel sound system for spatilalized audio rendering, and a vibro-mechanical platform to reproduce vibrosensory stimulus at the floor level. Unlike conventional videoconference environments, Shared Reality provides users with a sense of video and audio immersion in a common, shared space.  Remote users appear as if part of the local environment, and can manipulate synthetic (computer-generated) objects in the view.   To improve the quality of Shared Reality experience, this research now seeks to utilize additional cues available from input video in order provide faster, higher-resolution displays that do not suffer from shadows cast by users who block the projector output, and can render views that appropriate to the current participant position(s).  Some of the techniques employed include image mosaicing, robust tracking and gesture acquisition, dynamic background segmentation, and depth estimation from stereo.   Applied together, this research has the potential to support the demands of distributed immersive visualization, such as required by groups of engineers working collaboratively on a complex problem that requires interaction both with each other and the synthetic model.