SCI: SGER: Collaborative Research: Experimentation with TEEVE Tele-Immersive Infrastructure

远程沉浸是一种新兴的技术，使用户的互动，从事活动，如训练太极运动，舞蹈，或协助物理治疗，地理分布的地点之间。这是通过在真实的时间中三维（3D）空间中的活动的逼真视频和声音重建来实现的。这些远程沉浸式环境的几个组件需要到位，以实现无缝的沉浸式体验：（a）覆盖活动正在发生的360度空间的摄像机组，因此创建3D数据集，（B）从360度恢复声音而没有回声的声音系统，（c）宽带网络技术，可在地理上分散的地点提供大量数据集（d）以一致的方式显示来自不同站点的数据的显示技术。已经开发了远程沉浸式环境的许多组件，并且已经进行了具有个体和定制组件的初步小规模实验。缺少的是对当前现有远程沉浸式组件的整体部署和实验。在这个为期一年的项目中，我们的目标是（1）整合目前市场上可用于远程沉浸式环境的COTS组件，例如当前的3D摄像头，无线音响系统，Internet2协议以及现有的等离子和投影仪型显示器，（2）在两个地理分布的站点部署集成的COTS组件，加州大学伯克利分校和伊利诺伊大学香槟分校（UIUC），以及（3）使用我们的整体集成系统测试和执行初步实验，该系统被称为每个人的远程沉浸式环境（TEEVE）。这种集成和部署将带来非常独特的结果，因为它们将能够测试现有的COTS组件及其功能和局限性，从而为更广泛的受众提供尖端技术。我们还预计，集成和部署将揭示未来的研究挑战，因为我们计划进行广泛的实验分析的TEEVE系统。我们的实验分析将包括基于网络的测量，我们计划在发送大量音频和视频流时测量和评估Internet 2上完成的带宽，完成的延迟，抖动，丢失率，响应时间，实时3D数据的压缩比，基于视觉的测量，我们计划在语义和实时行为方面测量现有3D重建算法的正确性，到基于用户的测量，我们计划在UC Berkeley一侧与太极老师进行实验，并在UIUC一侧询问学生关于他们使用标准录像带的经验。智力上的优点在于（1）COTS组件的广泛集成，它将考虑和分析我们多层系统中可能的多种配置和同步机制，（2）集成系统的部署和实验，测量不同系统/用户级别的各种指标，以及（3）集成指南和设置方法，如何将这样一个整体的远程沉浸式基础设施（如TEEVE）放在一起。经验和实验结果将在ACM Multimedia、IEEE International Conference on Multimedia and Expo（ICME）等会议上以文档、软件和教程的形式与研究界分享。这将是第一个全面部署可负担得起的远程沉浸式互联网基础设施的项目，将产生更广泛的影响。TEEVE是允许跨地理分布的站点共享需要全身视觉和听觉信息的活动的第一步。实例存在于诸如对远程位置的用户进行物理治疗的健康护理中（例如，印第安人，爱斯基摩人），在训练的体育活动，如太极，或在艺术培训，如舞蹈。

Tele-immersion is an emerging technology that enables interaction of users, engaged in activities such as training of tai-chi movements, dancing, or assistance in physical therapy, between geographically distributed sites. This is achieved through realistic video and sound reconstruction of the activities in three-dimensional (3D) space in real time. Several components of these tele-immersive environments need to be in place to achieve seamless immersive experience: (a) set of cameras covering the 360 degree space in which the activity is taking place, hence creating 3D data sets, (b) sound system recovering sound from 360 degrees without echo, (c) broadband networking technology that enables throughput of large data sets across geographically distributed sites (end-to-end) with minimal latency and synchronous delivery, and (d) display technologies to display data from different sites in a consistent fashion. Many components of the tele-immersive environments have been developed, and preliminary small scale experiments with individual and customized components have been performed. What is missing is a holistic deployment and experimentation with the current existing tele- immersive components. In this one-year project, we aim to (1) integrate current COTS components that are available on the market for tele-immersive environments, such as the current 3D cameras, wireless sound system, Internet2 protocols, and existing plasma and projector-type displays, (2) deploy the integrated COTS components at two geographically distributed sites, UC Berkeley and University of Illinois at Urbana-Champaign (UIUC), and (3) test and execute preliminary experiments with our holistic integrated system, called Tele-immersive Environment for EVErybody (TEEVE). This integration and deployment will present very unique results because they will enable testing of the existing COTS components and their capabilities and limitations to deliver cutting-edge technology to broader audience. We expect also that the integration and deployment will reveal future research challenges since we plan to perform extensive experimental analysis of the TEEVE system. Our experimental analysis will span from network-based measurements where we plan to measure and evaluate accomplished bandwidth across Internet2 when sending large number of audio and video streams, accomplished latency, jitter, loss rate, response time, compression ratio on real-time 3D data, vision-based measurements where we plan to measure the correctness of existing 3D reconstruction algorithms in terms of semantics and real-time behavior, to user-based measurements where we plan to experiment with the tai-chi teacher at the UC Berkeley side and question students at the UIUC side about their experience versus using a standard video tape.The intellectual Merit rests in (1) the extensive integration of COTS components which will consider and analyze the multiple configurations and synchronization mechanisms possible in our multi-tier system, (2) the deployment and experimentation with the integrated system, measuring the various metrics at different system/user levels, and (3) integration guidelines and setup methodologies how to put together such a holistic tele-immersive infrastructure like TEEVE. Experiences and experimental results will be shared with the research community in the form of documentation, software, and tutorial(s) at conferences such as ACM Multimedia, IEEE International Conference on Multimedia and Expo (ICME) and others.Broader impacts will derive from the fact that will be the first holistic deployment of an affordable tele-immersive Internet-based infrastructure. TEEVE is the first step to allow sharing of activities that require full body visual and auditory information across geographically distributed sites. Examples exist in health care such as physical therapy to users in remote locations (e.g., Indians, Eskimos), in training of sport activities such as tai-chi, or in art training such as dancing.