NetSE:Large:Collaborative Research: Exploiting Multi-modality for Tele-Immersion

提供一个既能让人沉浸其中又能让人互动的环境是一项艰巨的研究挑战。在使用地理分布的城市中安装的这些环境时，确保合理的体验质量（QoE）是一项更艰巨的挑战。这个项目考虑了一个协作的、沉浸式的、互动的环境，不仅支持参与者的3D渲染。视频，还有其他方式，如身体传感器网络（BSN）数据，可以提供关于一个人的高度精确的数据？S的身体运动（以及生理数据）。在创建此环境时，需要考虑阻塞承载沉浸式交互式信息的数据流的各种瓶颈：重构延迟、所需的超高吞吐量、数据包丢失和呈现延迟。该项目的主要目标是设计和开发具有更高帧速率和帧质量的协作式多模式沉浸式环境，通过开展研究任务，利用来自其他模式的信息并处理这些瓶颈。在典型的远程沉浸式环境中，参与者可以在本地渲染的3D视图中看到自己，也可以在远程环境中看到参与者。由于本地呈现延迟要小得多，与遭受通信延迟和数据包丢失的远程参与者的呈现相比，参与者可以更早地以更流畅的方式看到自己。沉浸式参与者之间的这种不同延迟可能会在动态交互过程中导致问题，并影响他们的QoE。诸如可能导致什么类型的问题以及参与者如何处理这些问题等问题的答案取决于沉浸式环境的应用领域。为了研究QoE并验证（通过可用性研究）协作式沉浸式环境，将在多个城市部署远程康复应用程序：加州伯克利；2个地点在德克萨斯州的达拉斯；以及伊利诺斯州的厄巴纳-香槟。本项目的智力优势是：(i)多源、多目的地、多速率、多模态数据流的资源适应框架结合了基于监督混合控制理论的细粒度资源管理、多模态粗粒度管理和多模态多播方法。（ii）基于图形处理单元（GPU）的3D重建和压缩算法。这些算法有助于基于3D相机阵列数据的3D数据点重建，并以比基于cpu的同行更快的速度压缩它们。（iii）基于gpu的接收端3D数据渲染算法。该算法将利用BSN数据流中的骨架信息处理3D相机数据流中潜在的数据丢失。（iv）识别和测量体验质量（QoE）指标，并使用这些指标推导服务质量（QoS）参数。派生的QoS参数将帮助资源适应框架在运行时修改其决策。该项目旨在在利用多模态的新算法集中具有变革性的方面，同时将基于体验质量的反馈纳入流媒体、3D重建和渲染等功能。更广泛的影响：该项目通过提高执行复杂项目的能力，如提高准确性和灵活性的远程康复，有望在教育和普及卫生保健领域产生重大影响。考虑到保健专业人员可能处理更多人口（在偏远地区）的能力，以及考虑到受影响的人更快地独立和生产的可能性，这也可以导致社会生产力的提高。该项目还确保拟议研究的结果将纳入所教授的课程。3名女博士生和6名本科生（2名少数民族学生）已经与该项目的研究人员一起工作。将作出认真努力，使它们继续参与这个项目。除了经过评审的会议和期刊出版物外，开发的软件、收集的数据和研究成果将通过专门的网站与其他研究人员共享（在确保符合HIPAA规定的情况下）。

Providing an environment that offers both immersion and interaction is a tough research challenge. Ensuring a reasonable Quality of Experience (QoE) in using these environments installed in geographically distributed cities is even a tougher challenge. This project considers a collaborative, immersive, and interactive environment that not only supports 3D rendering of the participants? video but also other modalities such as Body Sensor Network (BSN) data that can offer highly precise data about a person?s physical movements (as well as physiological data). While creating this environment, one needs to consider the various bottlenecks that choke the data streams carrying the immersive and interactive information: reconstruction delay, ultra-high throughput needed, packet loss, and rendering delays. The main aim of this project is to design and develop collaborative, multi-modal immersive environments with higher frame rates and frame quality by carrying out research tasks that can take advantage of information from other modalities and handle these bottlenecks.In a typical tele-immersive environment, participants can see themselves in the locally rendered 3D view and see participants in the remote environments as well. Since the local rendering delays are much smaller, participants can see themselves earlier and in a more smooth fashion compared to the rendering of remote participants that suffers from communication delays and packet losses. This aspect of varying delays among the immersive participants can potentially cause problems during dynamic interactions and affect their QoE. Answers to questions such as what type of problems can be caused and how the participants handle them depend on the application domain of the immersive environments. To study the QoE and validate (with usability studies) the collaborative, immersive environment, a tele-rehabilitation application will be deployed in multiple cities: Berkeley, California; 2 sites in Dallas, Texas; and Urbana-Champaign, Illinois. Intellectual Merits of this project are (i) The resource adaptation framework for streaming multi-source, multi-destination, multi-rate, multi-modal data incorporates supervisory hybrid control theory based fine-grained resource management, multi-modal coarse-grained management, and a multi-modal multicasting approach. (ii) Graphics Processing Unit (GPU)-based 3D reconstruction and compression algorithms. These algorithms facilitate reconstruction of 3D data points based on 3D camera array data and compress them at a faster pace than their CPU-based counterparts. (iii) GPU-based rendering algorithm of 3D data on the receiver side. This algorithm will handle potential data loss in 3D camera data streams using skeletal information from BSN data streams. (iv) Identification and measurement of Quality of Experience (QoE) metrics and using those metrics to derive Quality of Service (QoS) parameters. The derived QoS parameters will then help the resource adaptation framework to modify its decisions at run-time. This project aims to have transformative aspects in the new set of algorithms that exploits multi-modality while incorporating a feedback based on Quality of Experience for functions such as streaming, 3D reconstruction, and rendering.Broader Impacts: This project promises significant impact in the fields of education and pervasive health care by providing augmented abilities to carry out intricate programs such as tele-rehabilitation with increased correctness and flexibility. This can also lead to improved productivity in the society considering the ability of health-care professionals to potentially handle a larger population (in remote places) as well as considering the possibility of the affected persons to become independent and productive faster. The project also ensures the results from the proposed research will be incorporated into the courses being taught. 3 women PhD students and 6 under-graduate students (2 are minority students) already working with the investigators of this project. Serious efforts will be undertaken to continue their involvement in this project. Apart from refereed conference and journal publications, the developed software, collected data, and research results will be shared with other researchers through a dedicated website (after ensuring satisfaction of HIPAA regulations).