NRI: FND: Communicate, Share, Adapt: A Mixed Reality Framework for Facilitating Robot Integration and Customization

为社交互动而设计的机器人可以通过多种方式丰富个人和社会的生活质量，例如自动化不受欢迎的体力工作，支持日常生活活动以及促进社会联系。然而，机器人与社会的无缝融合取决于人和机器人理解如何自然有效地相互沟通。人类需要了解机器人的能力，并建立对机器人的信任。机器人需要了解人类的能力和兴趣，并相应地调整自己的行为。该项目将使用虚拟现实和增强现实技术来帮助人和机器人在见面之前更好地了解对方。在共享虚拟教学体验（SVTE）期间，人类用户将能够使用虚拟现实和增强现实技术与虚拟版本的机器人进行交互。SVTE将帮助机器人和用户学习如何相互沟通，通过分享有关自己的信息建立信任和融洽关系，并根据彼此的学习进行调整。虽然SVTE是改善人机交互的一般方法，但该项目将重点关注老年人的需求，他们可能对使用和信任包括机器人在内的新技术犹豫不决。研究表明，老年人可以从与机器人的社交互动中受益，特别是当机器人帮助与其他老年人建立社交联系时。该项目有助于开发新的方法，利用虚拟现实和增强现实，帮助用户和机器人一起工作。它将使用社交辅助机器人来满足老龄化人口的社会需求，并推进人机交互的最新技术。 它还将涉及K-12学生，以激发他们对科学和工程的兴趣，同时在现实世界对以人为本的技术的需求背景下与老年社区互动。该项目使用虚拟现实（VR）和增强现实（AR）的沉浸式通信方式来克服阻碍协作机器人无缝集成到人类日常生活中的社交沟通障碍。具体来说，该项目正在开发一种沉浸式混合现实体验，即共享虚拟教学体验（SVTE），允许人类用户在真实的世界中进行交互之前习惯机器人，反之亦然。VR和AR都可能用于此目的，因为VR提供了一致的图形环境，使用户的通信清晰，而AR允许在真实的物理环境中与模拟机器人进行交互，同时受益于图形增强。该项目开发了VR和AR格式的SVTE预曝光，以实现沉浸式用户培训，了解如何与机器人进行通信，并了解其功能和情感限制。在整个SVTE交互过程中，系统将收集有关用户的信息，这些信息将允许机器人适应物理世界中的个性化交互。为了评估SVTE，在开发期间和完成后，该项目将进行一系列用户研究，首先是大学生，然后是老年人在高级生活设施。在对老年用户的SVTE评估中，这项工作将侧重于使用机器人协助社会参与和促进防止社会孤立，这已被证明会提高发病率和死亡率。 总的来说，该项目将包括开发一个混合现实开源测试平台，该平台能够通过ROS和机器人的虚拟通信策略与物理机器人进行通信。使用来自这些虚拟模态的数据，这项工作将生成通常难以在物理世界中感知和创建的多模态用户模型。反过来，这些模型将允许非专家了解如何有效地与物理机器人进行交互。这项研究将提供一个评估多模态用户模型的框架，用于在各种真实的世界环境中自然地进行通信，适应和个性化人机交互。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

Robots designed for social interactions can enrich the quality of life for individuals and society in a myriad of ways, such as automating undesirable physical work, supporting activities of daily living, and facilitating social connections. However, seamless integration of robots into society depends on both people and robots understanding how to communicate naturally and effectively with each other. Humans need to understand the capabilities of a robot and develop trust in the robot. Robots need to understand the capabilities and interests of the human and adapt their behavior accordingly. This project will use virtual reality and augmented reality technologies to help people and robots to better understand each other before they meet in person. During a Shared Virtual Teaching Experience (SVTE), human users will be able to interact with a virtual version of a robot using virtual reality and augmented reality technologies. The SVTE will help both the robot and the users learn how to communicate with each other, build trust and rapport by sharing information about themselves, and adapt based on what each learns from the other. While an SVTE is a general approach for improving human-robot interactions, this project will focus on the needs of older adults, who may be hesitant to use and trust new technologies, including robots. Research has shown that older adults can benefit from social interactions with robots, especially when the robots help to create social connections with other older adults. This project helps to develop new approaches, leveraging virtual reality and augmented reality, to help users and robots work together. It will address societal needs of an aging population using socially assistive robots and advance the state of the art in human-robot interaction. It will also involve K-12 students in order to stimulate interest in science and engineering while engaging with the elderly community in the context of a real-world need for human-centered technology.This project uses immersive communication modalities of virtual reality (VR) and augmented reality (AR) to overcome the barriers to social communication that hinder seamless integration of co-robots into human everyday lives. Specifically, the project is developing an immersive mixed reality experience, Shared Virtual Teaching Experience (SVTE), allows human users to get accustomed to robots, and vice versa, before interacting together in the real world. Both VR and AR are potentially useful for this purpose, since VR provides a consistent graphical environment that makes communication clear for users, while AR allows for situating the interaction with a simulated robot in a real, physical environment, while benefiting from graphical enhancements. This project develops SVTE pre-exposures in both VR and AR formats to enable immersive user training on how to communicate with the robot and to understand its functional and affective limitations. Throughout SVTE interactions, the system will collect information about users that will allow the robot to adapt for personalized interactions in the physical world. To evaluate the SVTE, during development and when completed, this project will perform a series of user studies, first with university students and then with older adults in a senior living facility. In the SVTE evaluation with elderly users, this work will focus on the use of robots for assisting in social engagement and facilitation for preventing social isolation, which has been shown to raise morbidity and mortality. Overall, this project will include the development of a mixed reality open-source testbed capable of communicating with physical robots via ROS and virtual communication strategies for robots. Using data from those virtual modalities, this work will generate multimodal user models that are typically difficult to perceive and create within the physical world. The models will, in turn, allow non-experts to understand how to effectively interact with physical robots. This research will provide a framework for evaluating multimodal user models for naturally communicating, adapting, and personalizing human-robot interactions in various real world contexts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.