NetSE: Large: Collaborative Research: Exploiting Multi-Modality for Tele-Immersion

提供既能让人身临其境又能让人互动的环境是一项艰巨的研究挑战。在使用安装在地理位置分散的城市的这些环境时，确保合理的体验质量(QOE)更是一项艰巨的挑战。这个项目考虑一个协作、身临其境和交互的环境，不仅支持参与者的3D渲染？不仅是视频，还包括其他方式，如身体传感器网络数据，可以提供关于一个人的高精度数据？S身体运动(以及生理数据)。在创建此环境时，需要考虑阻碍携带身临其境和交互信息的数据流的各种瓶颈：重建延迟、所需的超高吞吐量、丢包和渲染延迟。该项目的主要目标是通过执行可以利用其他模式的信息并解决这些瓶颈的研究任务，设计和开发具有更高帧速率和帧质量的协作、多模式沉浸式环境。在典型的远程沉浸式环境中，参与者可以在本地渲染的3D视图中看到自己，也可以在远程环境中看到参与者。由于本地渲染延迟要小得多，与遭受通信延迟和分组丢失的远程参与者的渲染相比，参与者可以更早、更流畅地看到自己。沉浸式参与者之间不同延迟的这一方面可能会在动态交互期间造成问题，并影响他们的QOE。诸如可能导致什么类型的问题以及参与者如何处理这些问题等问题的答案取决于沉浸式环境的应用领域。为了研究QOE并(通过可用性研究)验证协作、身临其境的环境，将在多个城市部署远程康复应用程序：加利福尼亚州伯克利；德克萨斯州达拉斯的2个地点；以及伊利诺伊州的Urbana-Champaign。该项目的智能优势在于：(1)多源、多目的地、多速率、多模式的流媒体资源适配框架融合了基于监督混合控制理论的细粒度资源管理、多模式粗粒度管理和多模式组播方法。(Ii)基于图形处理单元(GPU)的三维重建和压缩算法。这些算法促进了基于3D摄像机阵列数据的3D数据点的重建，并以比基于CPU的同行更快的速度压缩它们。(3)基于GPU的接收端3D数据渲染算法。该算法将使用来自BSN数据流的骨架信息来处理3D摄像机数据流中的潜在数据丢失。(Iv)识别和测量体验质量(QOE)度量，并使用这些度量来导出服务质量(QOS)参数。然后，导出的Qos参数将帮助资源适配框架在运行时修改其决策。该项目的目标是在利用多模式的新算法集中具有变革性的方面，同时结合基于流媒体、3D重建和渲染等功能的体验质量的反馈。广泛影响：该项目承诺通过提供增强的能力来执行复杂的程序，如远程康复，从而在教育和普遍的医疗保健领域产生重大影响，并提高正确性和灵活性。这还可以提高社会生产力，考虑到卫生保健专业人员可能处理更多人口(在偏远地区)的能力，以及考虑到受影响的人更快地独立和生产的可能性。该项目还确保拟议研究的结果将被纳入正在教授的课程。3名女博士生和6名本科生(2名是少数民族学生)已经与该项目的调查人员合作。将作出认真努力，继续参与这一项目。除了经评审的会议和期刊出版物外，开发的软件、收集的数据和研究结果将通过专门的网站与其他研究人员共享(在确保满足HIPAA规定后)。

Providing an environment that offers both immersion and interaction is a tough research challenge. Ensuring a reasonable Quality of Experience (QoE) in using these environments installed in geographically distributed cities is even a tougher challenge. This project considers a collaborative, immersive, and interactive environment that not only supports 3D rendering of the participants? video but also other modalities such as Body Sensor Network (BSN) data that can offer highly precise data about a person?s physical movements (as well as physiological data). While creating this environment, one needs to consider the various bottlenecks that choke the data streams carrying the immersive and interactive information: reconstruction delay, ultra-high throughput needed, packet loss, and rendering delays. The main aim of this project is to design and develop collaborative, multi-modal immersive environments with higher frame rates and frame quality by carrying out research tasks that can take advantage of information from other modalities and handle these bottlenecks.In a typical tele-immersive environment, participants can see themselves in the locally rendered 3D view and see participants in the remote environments as well. Since the local rendering delays are much smaller, participants can see themselves earlier and in a more smooth fashion compared to the rendering of remote participants that suffers from communication delays and packet losses. This aspect of varying delays among the immersive participants can potentially cause problems during dynamic interactions and affect their QoE. Answers to questions such as what type of problems can be caused and how the participants handle them depend on the application domain of the immersive environments. To study the QoE and validate (with usability studies) the collaborative, immersive environment, a tele-rehabilitation application will be deployed in multiple cities: Berkeley, California; 2 sites in Dallas, Texas; and Urbana-Champaign, Illinois. Intellectual Merits of this project are (i) The resource adaptation framework for streaming multi-source, multi-destination, multi-rate, multi-modal data incorporates supervisory hybrid control theory based fine-grained resource management, multi-modal coarse-grained management, and a multi-modal multicasting approach. (ii) Graphics Processing Unit (GPU)-based 3D reconstruction and compression algorithms. These algorithms facilitate reconstruction of 3D data points based on 3D camera array data and compress them at a faster pace than their CPU-based counterparts. (iii) GPU-based rendering algorithm of 3D data on the receiver side. This algorithm will handle potential data loss in 3D camera data streams using skeletal information from BSN data streams. (iv) Identification and measurement of Quality of Experience (QoE) metrics and using those metrics to derive Quality of Service (QoS) parameters. The derived QoS parameters will then help the resource adaptation framework to modify its decisions at run-time. This project aims to have transformative aspects in the new set of algorithms that exploits multi-modality while incorporating a feedback based on Quality of Experience for functions such as streaming, 3D reconstruction, and rendering.Broader Impacts: This project promises significant impact in the fields of education and pervasive health care by providing augmented abilities to carry out intricate programs such as tele-rehabilitation with increased correctness and flexibility. This can also lead to improved productivity in the society considering the ability of health-care professionals to potentially handle a larger population (in remote places) as well as considering the possibility of the affected persons to become independent and productive faster. The project also ensures the results from the proposed research will be incorporated into the courses being taught. 3 women PhD students and 6 under-graduate students (2 are minority students) already working with the investigators of this project. Serious efforts will be undertaken to continue their involvement in this project. Apart from refereed conference and journal publications, the developed software, collected data, and research results will be shared with other researchers through a dedicated website (after ensuring satisfaction of HIPAA regulations).