SCI: SGER: Collaborative Research: Experimentation with TEEVE Tele-Immersive Infrastructure

远程沉浸是一项新兴技术，它使用户能够在地理上分布的地点之间进行互动，参与诸如太极动作训练、舞蹈或协助理疗等活动。这是通过实时地对三维(3D)空间中的活动进行真实的视频和声音重建来实现的。这些远程沉浸式环境的几个组件需要到位以实现无缝的身临其境体验：(A)覆盖正在进行活动的360度空间的一组摄像机，从而创建3D数据集，(B)从360度恢复声音而没有回声的音响系统，(C)能够以最小的延迟和同步传输实现跨地理分布的站点(端到端)的大型数据集吞吐量的宽带网络技术，以及(D)以一致的方式显示来自不同站点的数据的显示技术。已经开发了远程沉浸式环境的许多组件，并对单个组件和定制组件进行了初步的小规模实验。缺少的是对当前现有的远程沉浸式组件的整体部署和试验。在这个为期一年的项目中，我们的目标是(1)集成目前市场上可用于远程沉浸式环境的COTS组件，例如当前的3D摄像机、无线音响系统、Internet2协议以及现有的等离子和投影仪类型显示器；(2)在两个地理位置分散的地点--加州大学伯克利分校和伊利诺伊大学香槟分校(UIUC)--部署集成的COTS组件；以及(3)利用我们的整体集成系统测试和执行初步实验，称为人人享有远程沉浸式环境(TEEVE)。这种集成和部署将呈现非常独特的结果，因为它们将能够测试现有的COTS组件及其向更广泛的受众提供尖端技术的能力和限制。我们还预计，集成和部署将揭示未来的研究挑战，因为我们计划对TEEVE系统进行广泛的实验分析。我们的实验分析将涵盖基于网络的测量，我们计划在发送大量音频和视频流时测量和评估Internet2上完成的带宽，完成的延迟、抖动、丢失率、响应时间、实时3D数据的压缩比，基于视觉的测量，我们计划在语义和实时行为方面测量现有3D重建算法的正确性，对于基于用户的测量，我们计划与加州大学伯克利分校的太极老师进行实验，并询问UIUC侧的学生与使用标准录像带的学生的体验。智力价值在于(1)COTS组件的广泛集成，它将考虑和分析我们的多层系统中可能出现的多种配置和同步机制，(2)集成系统的部署和实验，在不同系统/用户级别测量各种指标，以及(3)集成指南和设置方法如何组合这样一个完整的远程沉浸式基础设施，如TEEVE。经验和实验结果将在ACM多媒体、IEEE多媒体和博览会国际会议(ICME)等会议上以文档、软件和教程的形式与研究团体分享(S)。更广泛的影响将来自于这样一个事实，即首次全面部署负担得起的基于远程沉浸式互联网的基础设施。TEEVE是允许跨地理分布的地点共享需要全身视觉和听觉信息的活动的第一步。例如，对偏远地区(例如，印度人、爱斯基摩人)的使用者进行物理治疗的医疗保健，太极等体育活动的训练，或舞蹈等艺术训练。

Tele-immersion is an emerging technology that enables interaction of users, engaged in activities such as training of tai-chi movements, dancing, or assistance in physical therapy, between geographically distributed sites. This is achieved through realistic video and sound reconstruction of the activities in three-dimensional (3D) space in real time. Several components of these tele-immersive environments need to be in place to achieve seamless immersive experience: (a) set of cameras covering the 360 degree space in which the activity is taking place, hence creating 3D data sets, (b) sound system recovering sound from 360 degrees without echo, (c) broadband networking technology that enables throughput of large data sets across geographically distributed sites (end-to-end) with minimal latency and synchronous delivery, and (d) display technologies to display data from different sites in a consistent fashion. Many components of the tele-immersive environments have been developed, and preliminary small scale experiments with individual and customized components have been performed. What is missing is a holistic deployment and experimentation with the current existing tele- immersive components. In this one-year project, we aim to (1) integrate current COTS components that are available on the market for tele-immersive environments, such as the current 3D cameras, wireless sound system, Internet2 protocols, and existing plasma and projector-type displays, (2) deploy the integrated COTS components at two geographically distributed sites, UC Berkeley and University of Illinois at Urbana-Champaign (UIUC), and (3) test and execute preliminary experiments with our holistic integrated system, called Tele-immersive Environment for EVErybody (TEEVE). This integration and deployment will present very unique results because they will enable testing of the existing COTS components and their capabilities and limitations to deliver cutting-edge technology to broader audience. We expect also that the integration and deployment will reveal future research challenges since we plan to perform extensive experimental analysis of the TEEVE system. Our experimental analysis will span from network-based measurements where we plan to measure and evaluate accomplished bandwidth across Internet2 when sending large number of audio and video streams, accomplished latency, jitter, loss rate, response time, compression ratio on real-time 3D data, vision-based measurements where we plan to measure the correctness of existing 3D reconstruction algorithms in terms of semantics and real-time behavior, to user-based measurements where we plan to experiment with the tai-chi teacher at the UC Berkeley side and question students at the UIUC side about their experience versus using a standard video tape.The intellectual Merit rests in (1) the extensive integration of COTS components which will consider and analyze the multiple configurations and synchronization mechanisms possible in our multi-tier system, (2) the deployment and experimentation with the integrated system, measuring the various metrics at different system/user levels, and (3) integration guidelines and setup methodologies how to put together such a holistic tele-immersive infrastructure like TEEVE. Experiences and experimental results will be shared with the research community in the form of documentation, software, and tutorial(s) at conferences such as ACM Multimedia, IEEE International Conference on Multimedia and Expo (ICME) and others.Broader impacts will derive from the fact that will be the first holistic deployment of an affordable tele-immersive Internet-based infrastructure. TEEVE is the first step to allow sharing of activities that require full body visual and auditory information across geographically distributed sites. Examples exist in health care such as physical therapy to users in remote locations (e.g., Indians, Eskimos), in training of sport activities such as tai-chi, or in art training such as dancing.