SCI: SGER: Collaborative Research: Experimentation with TEEVE Tele-Immersive Infrastructure

远程沉浸是一种新兴技术，它使用户能够在地理分布的地点之间进行互动，从事诸如太极运动训练，舞蹈或物理治疗辅助等活动。这是通过在三维（3D）空间中实时重建活动的逼真视频和声音来实现的。这些远程沉浸式环境的几个组成部分需要到位，以实现无缝的沉浸式体验：(a)覆盖活动发生的360度空间的一组摄像机，从而创建3D数据集；(b)声音系统从360度无回声地恢复声音；(c)宽带网络技术，使大型数据集能够跨地理分布站点（端到端）以最小的延迟和同步传输；(d)显示技术，以一致的方式显示来自不同站点的数据。远程沉浸式环境的许多组件已经开发出来，并且已经使用个人和定制组件进行了初步的小规模实验。缺少的是对现有的远程沉浸式组件的全面部署和实验。在这个为期一年的项目中，我们的目标是(1)集成目前市场上可用于远程沉浸式环境的COTS组件，例如当前的3D摄像机，无线音响系统，Internet2协议以及现有的等离子和投影机类型显示器；(2)将集成的COTS组件部署在两个地理分布的站点，UC伯克利分校和伊利诺伊大学厄巴纳-香槟分校（UIUC）。(3)测试和执行我们的整体集成系统，称为人人远程沉浸式环境（TEEVE）的初步实验。这种集成和部署将呈现非常独特的结果，因为它们将能够测试现有的COTS组件及其能力和限制，从而向更广泛的受众提供尖端技术。我们还期望集成和部署将揭示未来的研究挑战，因为我们计划对TEEVE系统进行广泛的实验分析。我们的实验分析将涵盖基于网络的测量，我们计划测量和评估在发送大量音频和视频流时通过Internet2完成的带宽，完成延迟，抖动，损失率，响应时间，实时3D数据的压缩比，基于视觉的测量，我们计划测量现有3D重建算法在语义和实时行为方面的正确性。到以用户为基础的测量，我们计划与加州大学伯克利分校的太极老师进行实验，并向伊利诺伊大学伯克利分校的学生询问他们的体验，而不是使用标准录像带。智力优势在于：(1)广泛集成COTS组件，它将考虑和分析多层系统中可能的多种配置和同步机制，(2)集成系统的部署和实验，测量不同系统/用户级别的各种指标，以及(3)集成指南和设置方法，如何将这样一个整体的远程沉浸式基础设施（如TEEVE）放在一起。经验和实验结果将以文档、软件和教程的形式在ACM多媒体、IEEE国际多媒体会议和博览会（ICME）等会议上与研究界分享。更广泛的影响将来自于这一事实，这将是第一次全面部署一个可负担得起的远程沉浸式互联网基础设施。TEEVE是允许跨地理分布站点共享需要全身视觉和听觉信息的活动的第一步。这方面的例子存在于保健方面，如对边远地区用户（如印第安人、爱斯基摩人）的物理治疗，太极等体育活动的训练，或舞蹈等艺术训练。

Tele-immersion is an emerging technology that enables interaction of users, engaged in activities such as training of tai-chi movements, dancing, or assistance in physical therapy, between geographically distributed sites. This is achieved through realistic video and sound reconstruction of the activities in three-dimensional (3D) space in real time. Several components of these tele-immersive environments need to be in place to achieve seamless immersive experience: (a) set of cameras covering the 360 degree space in which the activity is taking place, hence creating 3D data sets, (b) sound system recovering sound from 360 degrees without echo, (c) broadband networking technology that enables throughput of large data sets across geographically distributed sites (end-to-end) with minimal latency and synchronous delivery, and (d) display technologies to display data from different sites in a consistent fashion. Many components of the tele-immersive environments have been developed, and preliminary small scale experiments with individual and customized components have been performed. What is missing is a holistic deployment and experimentation with the current existing tele- immersive components. In this one-year project, we aim to (1) integrate current COTS components that are available on the market for tele-immersive environments, such as the current 3D cameras, wireless sound system, Internet2 protocols, and existing plasma and projector-type displays, (2) deploy the integrated COTS components at two geographically distributed sites, UC Berkeley and University of Illinois at Urbana-Champaign (UIUC), and (3) test and execute preliminary experiments with our holistic integrated system, called Tele-immersive Environment for EVErybody (TEEVE). This integration and deployment will present very unique results because they will enable testing of the existing COTS components and their capabilities and limitations to deliver cutting-edge technology to broader audience. We expect also that the integration and deployment will reveal future research challenges since we plan to perform extensive experimental analysis of the TEEVE system. Our experimental analysis will span from network-based measurements where we plan to measure and evaluate accomplished bandwidth across Internet2 when sending large number of audio and video streams, accomplished latency, jitter, loss rate, response time, compression ratio on real-time 3D data, vision-based measurements where we plan to measure the correctness of existing 3D reconstruction algorithms in terms of semantics and real-time behavior, to user-based measurements where we plan to experiment with the tai-chi teacher at the UC Berkeley side and question students at the UIUC side about their experience versus using a standard video tape.The intellectual Merit rests in (1) the extensive integration of COTS components which will consider and analyze the multiple configurations and synchronization mechanisms possible in our multi-tier system, (2) the deployment and experimentation with the integrated system, measuring the various metrics at different system/user levels, and (3) integration guidelines and setup methodologies how to put together such a holistic tele-immersive infrastructure like TEEVE. Experiences and experimental results will be shared with the research community in the form of documentation, software, and tutorial(s) at conferences such as ACM Multimedia, IEEE International Conference on Multimedia and Expo (ICME) and others.Broader impacts will derive from the fact that will be the first holistic deployment of an affordable tele-immersive Internet-based infrastructure. TEEVE is the first step to allow sharing of activities that require full body visual and auditory information across geographically distributed sites. Examples exist in health care such as physical therapy to users in remote locations (e.g., Indians, Eskimos), in training of sport activities such as tai-chi, or in art training such as dancing.