Enhancing Human Abilities and Social Interaction in Context-Aware Pervasive Extended Reality

尽管最近的技术进步和公众对扩展现实（XR）领域的兴趣越来越大，但XR的实现仍然主要基于自中心和由自组织开环XR系统收集的零星信息。这限制了对动态和大规模XR环境中用户上下文和需求的全面理解。虚拟实体对用户的影响是有限的，因为在现实世界中固有的缺乏（或缺乏）物理和交互性，对个人和社会的影响在很大程度上没有得到充分的探索，尽管它具有及时的重要性。该项目旨在解决这些问题，并通过将XR与互补的尖端技术（如人工智能和泛在计算与物联网）相结合的融合研究，实现更有影响力的XR体验。在XR中，以智能和无处不在的虚拟实体形式体现的互补概念和技术将具有变革性。为了追求更具交互性的XR系统，更好地理解XR中的人类体验，以及XR在现实生活中的实际有效用例，本项目为XR建立了两个短期目标：(1)开发一个上下文感知的普及XR框架；(2)研究多模态（具体化）XR体验对增强人类能力和体验的影响。上下文感知的普及XR框架将以数字孪生范式的概念设计和开发集成新型传感和驱动平台，例如智能物联网设备和机器人单元，同时采用先进的机器学习技术来识别环境中的用户上下文。将开发涉及具身虚拟实体的各种交互式XR原型，并将采用人因科学的理论和科学知识，进行严格的用户研究，全面调查对人类能力和经验的影响。该研究在技术方面具有创新性，利用大量的多模态数据和先进的机器学习方法开发了一个全面的大规模智能XR环境，在科学方面也旨在揭示XR在人类能力和经验的不同维度上的影响。在包容和公平的研究和教育环境中培养多样化的高素质人才的同时，具有上下文感知的XR框架的技术发展和关于XR经验影响的形成性知识的研究成果将指导研究人员和实践者设计更具交流和协作性的XR系统。该研究对改善不同社会环境中的现有互动和学习机制具有重要影响，例如教育、协作以及医疗/工业模拟和培训，这将直接有利于加拿大XR产业和研究的未来。

Notwithstanding the recent technological advances and increasing public interests in the field of eXtended Reality (XR), the realization of XR is still largely based on egocentric and sporadic information collected by ad-hoc open-loop XR systems. This restricts a comprehensive understanding of the user's context and needs in the dynamic and large-scale XR environment. The influence of virtual entities over the users is limited due to the inherent lack (or absence) of physicality and interactivity in the real world, and the implications on individuals and society are largely under-explored despite the timely importance. This program tackles these issues and aims to realize more influential XR experience by conducting convergence research that combines XR with complementary cutting-edge technologies, e.g., Artificial Intelligence and Ubiquitous Computing with Internet of Things (IoT). The complementary concepts and technologies manifested in the form of intelligent and pervasive embodied virtual entities in XR would be transformative. While pursuing more interactive XR systems, better understanding of human experience in XR, and practically effective use cases of XR in real life, two short-term objectives are established for this program: (1) develop a context-aware pervasive XR framework, and (2) investigate the effects of multimodal (embodied) XR experience in enhancing human abilities and experience. The context-aware pervasive XR framework will be designed and developed integrating novel sensing and actuating platforms, e.g., smart IoT devices and robotic units, in a concept of digital twin paradigm, while employing advanced machine-learning techniques to recognize the user's context in the environment. Various interactive XR prototypes will be developed involving embodied virtual entities, and rigorous user studies will be conducted to comprehensively investigate the effects on human abilities and experience, adopting theories and scientific knowledge in human factors sciences. The research is novel in the technical aspect that employs a large volume of multimodal data and advanced machine-learning methods to develop a comprehensive large-scale smart XR environment, and also in the scientific aspect that aims to reveal the impacts of XR in different dimensions of human abilities and experience. While training a diverse group of highly qualified personnel in an inclusive and equitable research and education environment, the research outcomes with a technological development of context-aware XR framework and formative knowledge on the effects of XR experience will guide researchers and practitioners to design more communicative and collaborative XR systems. The research has significant impacts on improving existing interaction and learning mechanisms in different social settings, e.g., education, collaboration, and medical/industrial simulation and training, which will directly benefit the future of XR industry and research in Canada.