Collaborative Research: CNS Core: Medium: miVirtualSeat: Semantics-aware Content Distribution for Immersive Meeting Environments

COVID-19疫情大大加强了工作相关及社交环境中对虚拟会议的需求。它还强调了视频会议工具的严重缺乏，这些工具可以模拟面对面会议的丰富沉浸式体验，因为目前的工具经常导致由于必须通过非理想的通信媒介进行交互而引起的疲劳。 这个合作项目汇集了来自伊利诺伊大学厄巴纳-香槟分校、马萨诸塞州阿默斯特大学和新泽西理工学院的研究人员，研究、构建和评估一个名为miVirtualSeat的分布式系统，该系统更接近模拟面对面会议的沉浸式体验，包括物理会议空间中的物理和虚拟参与者。该项目专注于提供沉浸式会议体验的关键研究挑战，虚拟参与者在物理会议室中彼此交互。有些参与者实际上在物理会议室中，但实际上位于仅具有有限计算和网络资源的远程站点。挑战是（a）实时地检测、跟踪和定位联合沉浸式场景中的分布式物理和虚拟360度化身和对象，（B）减少传送集成和同步的360度、体积和2D/3D视频以及立体声音频的带宽和延迟，以及（c）以物理和虚拟参与者之间的自然交互的形式确保良好的体验质量。该项目解决了后大流行社会的一个紧迫而重要的需求，即在教室、会议、办公室会议和社交聚会的背景下实现沉浸式混合会议。miVirtualSeat将通过位于每个研究机构的物理会议室和远程站点实现这些会议。该项目的成果将是先进混合现实沉浸式环境新兴领域的新本科生和研究生课程。通过外展活动，项目成员将展示miVirtualSeat，并向更广泛的公众展示分布式AR/VR系统的功能。该项目的信息和结果将保存在向公众提供的项目网站上：https://monet.cs.illinois.edu/miVirtualSeat/。这些将包括出版物，链接到软件存储库，媒体发布，并链接到系统数据存储库从项目派生。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

The COVID-19 pandemic has greatly reinforced the need for virtual meetings in both work-related and social settings. It has also highlighted the dire lack of video conferencing tools that can simulate the rich immersive experience of in-person meetings since the current tools often leading to fatigue caused by having to interact over a non-ideal communication medium. This collaborative project brings together investigators from University of Illinois, Urbana-Champaign, University of Massachusetts at Amherst, and New Jersey Institute of Technology to research, build, and evaluate a distributed system, called miVirtualSeat, that more closely simulates the immersive experience of in-person meetings, including physical and virtual participants in a physical meeting space.The project is focused on key research challenges in providing an immersive meeting experience where physical and virtual participants interact with each other in a physical meeting room. Some participants are virtually present in the physical meeting room, but physically located in remote sites with only limited compute and network resources. The challenges are (a) detecting, tracking, and localizing distributed physical and virtual 360-degree avatars and objects in the joint immersive scene in real-time, (b) reducing the bandwidth and latency of delivering integrated and synchronized 360-degree, volumetric, and 2D/3D video, and ambisonics audio, and (c) ensuring good quality-of-experience in the form of natural interaction between physical and virtual participants. The project addresses an immediate and important need for a post-pandemic society to enable immersive hybrid meetings that arise in the context of classrooms, conferences, office meetings, and social gatherings. miVirtualSeat will enable these meetings with a physical meeting room and remote sites situated at each of the investigators’ institutions. The outcome of the project will be new undergraduate and graduate courses in the emerging field of advanced mixed reality immersive environments. Through outreach activities, the project members will showcase miVirtualSeat and expose the broader public to the capabilities of distributed AR/VR systems.Information and results from this project will be maintained on the project website that is available to the public at https://monet.cs.illinois.edu/miVirtualSeat/. These will include publications, links to software repositories, media releases, and links to system data repositories derived from the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.