Collaborative Research: CNS Core: Medium: miVirtualSeat: Semantics-aware Content Distribution for Immersive Meeting Environments

2019冠状病毒病大流行极大地增加了在工作和社交环境中进行虚拟会议的需求。它还强调了视频会议工具的严重缺乏，这些工具可以模拟丰富的身临其境的面对面会议体验，因为目前的工具经常导致疲劳，因为必须在非理想的通信媒介上进行交互。这个合作项目汇集了来自伊利诺伊大学厄巴纳-香槟分校、马萨诸塞大学阿默斯特分校和新泽西理工学院的研究人员，研究、构建和评估一个名为miVirtualSeat的分布式系统，该系统更接近地模拟了面对面会议的沉浸式体验，包括物理会议空间中的物理和虚拟参与者。该项目专注于提供沉浸式会议体验的关键研究挑战，物理和虚拟参与者在物理会议室中相互互动。有些参与者实际上在物理会议室中，但物理上位于只有有限计算和网络资源的远程站点。挑战是(a)实时检测、跟踪和定位联合沉浸场景中的分布式物理和虚拟360度角色和对象，(b)减少提供集成和同步的360度、体积、2D/3D视频和立体声音频的带宽和延迟，以及(c)确保物理和虚拟参与者之间自然交互形式的良好体验质量。该项目解决了大流行后社会的迫切和重要需求，即在教室、会议、办公室会议和社交聚会的背景下实现沉浸式混合会议。miVirtualSeat将为这些会议提供物理会议室和位于每个调查机构的远程站点。该项目的成果将是在新兴的先进混合现实沉浸式环境领域开设新的本科和研究生课程。通过外展活动，项目成员将展示miVirtualSeat，并向更广泛的公众展示分布式AR/VR系统的功能。该项目的信息和结果将保存在该项目的网站上，公众可以访问https://monet.cs.illinois.edu/miVirtualSeat/。这些将包括出版物、到软件存储库的链接、媒体发布，以及到来自项目的系统数据存储库的链接。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The COVID-19 pandemic has greatly reinforced the need for virtual meetings in both work-related and social settings. It has also highlighted the dire lack of video conferencing tools that can simulate the rich immersive experience of in-person meetings since the current tools often leading to fatigue caused by having to interact over a non-ideal communication medium. This collaborative project brings together investigators from University of Illinois, Urbana-Champaign, University of Massachusetts at Amherst, and New Jersey Institute of Technology to research, build, and evaluate a distributed system, called miVirtualSeat, that more closely simulates the immersive experience of in-person meetings, including physical and virtual participants in a physical meeting space.The project is focused on key research challenges in providing an immersive meeting experience where physical and virtual participants interact with each other in a physical meeting room. Some participants are virtually present in the physical meeting room, but physically located in remote sites with only limited compute and network resources. The challenges are (a) detecting, tracking, and localizing distributed physical and virtual 360-degree avatars and objects in the joint immersive scene in real-time, (b) reducing the bandwidth and latency of delivering integrated and synchronized 360-degree, volumetric, and 2D/3D video, and ambisonics audio, and (c) ensuring good quality-of-experience in the form of natural interaction between physical and virtual participants. The project addresses an immediate and important need for a post-pandemic society to enable immersive hybrid meetings that arise in the context of classrooms, conferences, office meetings, and social gatherings. miVirtualSeat will enable these meetings with a physical meeting room and remote sites situated at each of the investigators’ institutions. The outcome of the project will be new undergraduate and graduate courses in the emerging field of advanced mixed reality immersive environments. Through outreach activities, the project members will showcase miVirtualSeat and expose the broader public to the capabilities of distributed AR/VR systems.Information and results from this project will be maintained on the project website that is available to the public at https://monet.cs.illinois.edu/miVirtualSeat/. These will include publications, links to software repositories, media releases, and links to system data repositories derived from the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.