Collaborative Research: CSR: Medium: Adaptive Environmental Awareness for Collaborative Augmented Reality

增强现实（AR）代表了移动应用的一个令人兴奋的新领域，它融合了物理世界和虚拟世界，促进了用户与周围物理环境之间的实时交互。下一代AR以交互式和环境感知应用为中心，通过计算量大的感知算法感知现实世界的状态，并为多个用户实时增强结果。然而，人们普遍认为，当前具有环境意识的AR应用程序缺乏足够的弹性，无法进行长期、大规模和安全关键型的部署。该项目的目标是利用下一代AR应用程序的协作特性，提供可靠、实时的环境意识，以适应周围环境、用户移动性和计算资源可用性。该项目的核心观点是，AR设备可以相互协作，运行和调整AR应用程序，以优化整体效用，而不是简单地相互竞争网络和计算资源。该项目将涉及对AR中的环境意识和移动模式进行建模，将这些模型整合到模拟器中以促进实验和培训，并设计强化学习策略，以捕获环境意识和动态用户关系。这些政策将部署在一系列新的协作AR应用程序中，例如基于AR的物理康复，并通过在广泛条件下对多组用户进行广泛研究来评估。该项目将为有弹性的下一代AR应用奠定基础，这将使零售、制造业和医疗保健等多个领域受益。该项目将开发下一代ar支持的康复应用程序示例，该应用程序将向多个医院系统的临床医生展示，并将整合到pi的K-12和公共推广工作中。作为该项目的一部分创建的多用户AR模拟器将是开源的，以便开发广泛的其他应用程序，并将被设计为可扩展到其他用例，例如AR辅助人机协作。此外，作为该项目一部分开发的用于动态用户协作的机器学习技术将适用于多个实体以协调方式动态交互的其他情况，例如供应链管理。该项目将培训几组本科生，其成果将在多个活动和场所展示，旨在扩大计算机的参与。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Augmented reality (AR) represents an exciting new frontier for mobile applications, merging the physical and virtual worlds to facilitate real-time interactions between users and their surrounding physical environments. Next-generation AR is centered on interactive and environmentally-aware applications, which sense the state of the real world via computationally heavy perception algorithms, with results augmented in real-time for multiple users. However, current environmentally-aware AR applications are well-recognized to lack sufficient resilience for long-term, large-scale, and safety-critical deployments. The goal of this project is to exploit the collaborative nature of next-generation AR applications to provide reliable, real-time environmental awareness that can adapt to the surrounding environment, user mobility, and computing resource availability. The core insight of this project is that instead of simply competing with one another for network and compute resources, AR devices can collaborate with one another to run and adapt AR applications to optimize overall utility. The project will involve modeling environmental awareness and mobility patterns in AR, incorporating these models into an emulator to facilitate experimentation and training, and designing reinforcement learning policies that capture environmental awareness and dynamic user relationships. These policies will be deployed in a set of new collaborative AR applications, such as AR-based physical rehabilitation, and evaluated via expansive studies with multiple groups of users in a wide range of conditions. This project will lay the groundwork for resilient next-generation AR applications, which can benefit multiple domains, including retail, manufacturing, and healthcare. The project will develop an example next-generation AR-supported rehabilitation application, which will be showcased to clinicians across multiple hospital systems and will be integrated into the PIs’ K-12 and public outreach efforts. The multi-user AR emulator created as part of this project will be open-sourced to enable the development of a wide range of other applications and will be designed to be extensible to additional use cases, such as AR-aided human-robot collaboration. Moreover, the machine learning techniques for dynamic user collaboration developed as part of this project will be applicable to other cases in which multiple entities interact dynamically in a coordinated manner, such as supply chain management. The project will train several cohorts of undergraduate students, and results will be presented at multiple events and venues aimed at broadening participation in computing.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.