NetSE:Large:Collaborative Research: Exploiting Multi-modality for Tele-Immersion

NetSE：大型：协作研究：利用多模态实现远程沉浸

• 批准号: 1012975
• 负责人: Balakrishnan Prabhakaran
• 金额: $ 203.38万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2010
• 资助国家: 美国
• 起止时间: 2010-10-01 至 2018-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1012975&HistoricalAwards=false
• 关键词: NetSE; Large; Collaborative; Research; Exploiting

Providing an environment that offers both immersion and interaction is a tough research challenge. Ensuring a reasonable Quality of Experience (QoE) in using these environments installed in geographically distributed cities is even a tougher challenge. This project considers a collaborative, immersive, and interactive environment that not only supports 3D rendering of the participants? video but also other modalities such as Body Sensor Network (BSN) data that can offer highly precise data about a person?s physical movements (as well as physiological data). While creating this environment, one needs to consider the various bottlenecks that choke the data streams carrying the immersive and interactive information: reconstruction delay, ultra-high throughput needed, packet loss, and rendering delays. The main aim of this project is to design and develop collaborative, multi-modal immersive environments with higher frame rates and frame quality by carrying out research tasks that can take advantage of information from other modalities and handle these bottlenecks.In a typical tele-immersive environment, participants can see themselves in the locally rendered 3D view and see participants in the remote environments as well. Since the local rendering delays are much smaller, participants can see themselves earlier and in a more smooth fashion compared to the rendering of remote participants that suffers from communication delays and packet losses. This aspect of varying delays among the immersive participants can potentially cause problems during dynamic interactions and affect their QoE. Answers to questions such as what type of problems can be caused and how the participants handle them depend on the application domain of the immersive environments. To study the QoE and validate (with usability studies) the collaborative, immersive environment, a tele-rehabilitation application will be deployed in multiple cities: Berkeley, California; 2 sites in Dallas, Texas; and Urbana-Champaign, Illinois. Intellectual Merits of this project are (i) The resource adaptation framework for streaming multi-source, multi-destination, multi-rate, multi-modal data incorporates supervisory hybrid control theory based fine-grained resource management, multi-modal coarse-grained management, and a multi-modal multicasting approach. (ii) Graphics Processing Unit (GPU)-based 3D reconstruction and compression algorithms. These algorithms facilitate reconstruction of 3D data points based on 3D camera array data and compress them at a faster pace than their CPU-based counterparts. (iii) GPU-based rendering algorithm of 3D data on the receiver side. This algorithm will handle potential data loss in 3D camera data streams using skeletal information from BSN data streams. (iv) Identification and measurement of Quality of Experience (QoE) metrics and using those metrics to derive Quality of Service (QoS) parameters. The derived QoS parameters will then help the resource adaptation framework to modify its decisions at run-time. This project aims to have transformative aspects in the new set of algorithms that exploits multi-modality while incorporating a feedback based on Quality of Experience for functions such as streaming, 3D reconstruction, and rendering.Broader Impacts: This project promises significant impact in the fields of education and pervasive health care by providing augmented abilities to carry out intricate programs such as tele-rehabilitation with increased correctness and flexibility. This can also lead to improved productivity in the society considering the ability of health-care professionals to potentially handle a larger population (in remote places) as well as considering the possibility of the affected persons to become independent and productive faster. The project also ensures the results from the proposed research will be incorporated into the courses being taught. 3 women PhD students and 6 under-graduate students (2 are minority students) already working with the investigators of this project. Serious efforts will be undertaken to continue their involvement in this project. Apart from refereed conference and journal publications, the developed software, collected data, and research results will be shared with other researchers through a dedicated website (after ensuring satisfaction of HIPAA regulations).

提供一个既能提供沉浸感又能提供互动的环境是一个坚韧的研究挑战。确保在使用安装在地理上分散的城市中的这些环境时获得合理的体验质量（QoE）甚至是一个更严峻的挑战。该项目认为，一个协作，沉浸式和交互式的环境，不仅支持3D渲染的参与者？视频，以及其他形式，如身体传感器网络（BSN）数据，可以提供关于一个人的高度精确的数据？的物理运动（以及生理数据）。在创建此环境时，需要考虑阻塞承载沉浸式和交互式信息的数据流的各种瓶颈：重建延迟、所需的超高吞吐量、数据包丢失和渲染延迟。该项目的主要目的是设计和开发具有更高帧速率和帧质量的协作多模态沉浸式环境，通过执行可以利用其他模态信息并处理这些瓶颈的研究任务。在典型的远程沉浸式环境中，参与者可以在本地渲染的3D视图中看到自己，也可以在远程环境中看到参与者。由于本地渲染延迟要小得多，与遭受通信延迟和分组丢失的远程参与者的渲染相比，参与者可以更早地并且以更平滑的方式看到自己。沉浸式参与者之间的不同延迟的这一方面可能在动态交互期间引起问题并影响他们的QoE。诸如可能导致什么类型的问题以及参与者如何处理这些问题等问题的答案取决于沉浸式环境的应用领域。为了研究QoE并验证（使用可用性研究）协作、沉浸式环境，将在多个城市部署远程康复应用程序：加州的伯克利;德克萨斯州达拉斯的2个站点;以及伊利诺伊州的厄巴纳-香槟。该项目的智能优点是：（i）用于流传输多源、多目的地、多速率、多模态数据的资源自适应框架结合了基于监督混合控制理论的细粒度资源管理、多模态粗粒度管理和多模态多播方法。(ii)基于图形处理单元（GPU）的3D重建和压缩算法。这些算法有助于基于3D相机阵列数据重建3D数据点，并以比基于CPU的对应方更快的速度压缩它们。(iii)接收端基于GPU的3D数据渲染算法。该算法将使用来自BSN数据流的骨架信息来处理3D相机数据流中的潜在数据丢失。(iv)识别和测量体验质量（QoE）度量，并使用这些度量导出服务质量（QoS）参数。然后，导出的QoS参数将帮助资源自适应框架在运行时修改其决策。该项目旨在为新的算法集带来变革性的方面，该算法集利用多模态，同时将基于体验质量的反馈纳入流媒体、3D重建和渲染等功能。该项目通过提供增强的能力来执行复杂的方案，如远程教育，康复与增加的正确性和灵活性。考虑到卫生保健专业人员有可能处理更多人口（在偏远地区）的能力，并考虑到受影响者有可能更快地独立和生产，这也可以提高社会的生产力。该项目还确保拟议研究的结果将纳入正在教授的课程。3名女博士生和6名本科生（2名是少数民族学生）已经与该项目的研究人员一起工作。将作出认真努力，继续参与这一项目。除了参考会议和期刊出版物外，开发的软件，收集的数据和研究结果将通过专门的网站与其他研究人员共享（在确保满足HIPAA法规之后）。