Collaborative Research: CNS Core: Medium: miVirtualSeat: Semantics-aware Content Distribution for Immersive Meeting Environments

新冠肺炎的流行极大地增强了人们在工作和社交环境中使用虚拟会议的需求。它还突显了严重缺乏能够模拟丰富的身临其境的面对面会议体验的视频会议工具，因为当前的工具往往会导致疲惫，因为不得不通过非理想的交流媒介进行交互。这个合作项目汇集了来自伊利诺伊大学、厄巴纳-香槟分校、马萨诸塞大学阿默斯特分校和新泽西理工学院的研究人员，研究、构建和评估一个名为miVirtualSeat的分布式系统，该系统更接近于模拟面对面会议的身临其境的体验，包括物理会议空间中的物理和虚拟参与者。该项目专注于在提供身临其境的会议体验方面的关键研究挑战，其中物理和虚拟参与者在物理会议室中相互交互。有些与会者虚拟地出现在实体会议室中，但实际位于仅有有限计算和网络资源的远程站点。挑战在于：(A)实时检测、跟踪和定位联合沉浸式场景中的分布式物理和虚拟360度化身和对象，(B)减少提供集成和同步的360度、体积和2D/3D视频以及两性音频的带宽和延迟，以及(C)以物理和虚拟参与者之间自然交互的形式确保良好的体验质量。该项目解决了大流行后社会的迫切和重要需求，以使在教室、会议、办公室会议和社交聚会中出现的身临其境的混合会议成为可能。MiVirtualSeat将使这些会议有一个实体会议室和位于每个调查机构的远程地点。该项目的成果将是在先进的混合现实沉浸式环境这一新兴领域开设新的本科生和研究生课程。通过外展活动，项目成员将展示miVirtualSeat，并向更广泛的公众展示分布式AR/VR系统的能力。该项目的信息和结果将保存在项目网站上，供公众访问https://monet.cs.illinois.edu/miVirtualSeat/.这些奖励将包括出版物、软件资源库的链接、媒体发布和从该项目派生的系统数据资源库的链接。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The COVID-19 pandemic has greatly reinforced the need for virtual meetings in both work-related and social settings. It has also highlighted the dire lack of video conferencing tools that can simulate the rich immersive experience of in-person meetings since the current tools often leading to fatigue caused by having to interact over a non-ideal communication medium. This collaborative project brings together investigators from University of Illinois, Urbana-Champaign, University of Massachusetts at Amherst, and New Jersey Institute of Technology to research, build, and evaluate a distributed system, called miVirtualSeat, that more closely simulates the immersive experience of in-person meetings, including physical and virtual participants in a physical meeting space.The project is focused on key research challenges in providing an immersive meeting experience where physical and virtual participants interact with each other in a physical meeting room. Some participants are virtually present in the physical meeting room, but physically located in remote sites with only limited compute and network resources. The challenges are (a) detecting, tracking, and localizing distributed physical and virtual 360-degree avatars and objects in the joint immersive scene in real-time, (b) reducing the bandwidth and latency of delivering integrated and synchronized 360-degree, volumetric, and 2D/3D video, and ambisonics audio, and (c) ensuring good quality-of-experience in the form of natural interaction between physical and virtual participants. The project addresses an immediate and important need for a post-pandemic society to enable immersive hybrid meetings that arise in the context of classrooms, conferences, office meetings, and social gatherings. miVirtualSeat will enable these meetings with a physical meeting room and remote sites situated at each of the investigators’ institutions. The outcome of the project will be new undergraduate and graduate courses in the emerging field of advanced mixed reality immersive environments. Through outreach activities, the project members will showcase miVirtualSeat and expose the broader public to the capabilities of distributed AR/VR systems.Information and results from this project will be maintained on the project website that is available to the public at https://monet.cs.illinois.edu/miVirtualSeat/. These will include publications, links to software repositories, media releases, and links to system data repositories derived from the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.