NRI: FND: Communicate, Share, Adapt: A Mixed Reality Framework for Facilitating Robot Integration and Customization

为社会互动而设计的机器人可以以无数种方式丰富个人和社会的生活质量，例如自动化不受欢迎的体力工作，支持日常生活活动，促进社会联系。然而，机器人无缝融入社会取决于人和机器人都了解如何自然有效地相互沟通。人类需要了解机器人的能力，培养对机器人的信任。机器人需要了解人类的能力和兴趣，并相应地调整它们的行为。这个项目将使用虚拟现实和增强现实技术来帮助人和机器人在见面之前更好地了解对方。在共享虚拟教学体验（SVTE）中，人类用户将能够使用虚拟现实和增强现实技术与虚拟版本的机器人进行交互。SVTE将帮助机器人和用户学习如何相互沟通，通过分享自己的信息建立信任和融洽关系，并根据彼此的学习情况进行调整。虽然SVTE是改善人机交互的通用方法，但该项目将重点关注老年人的需求，他们可能对使用和信任包括机器人在内的新技术犹豫不决。研究表明，老年人可以从与机器人的社交互动中受益，尤其是当机器人帮助与其他老年人建立社交联系时。该项目有助于开发利用虚拟现实和增强现实的新方法，以帮助用户和机器人协同工作。它将解决使用社交辅助机器人的老龄化人口的社会需求，并推进人机交互的最新技术。它还将涉及K-12学生，以激发他们对科学和工程的兴趣，同时在现实世界对以人为本的技术需求的背景下，与老年人社区合作。该项目使用虚拟现实（VR）和增强现实（AR）的沉浸式通信方式来克服阻碍协作机器人无缝融入人类日常生活的社交沟通障碍。具体来说，该项目正在开发一种沉浸式混合现实体验，即共享虚拟教学体验（SVTE），允许人类用户习惯机器人，反之亦然，然后在现实世界中一起互动。VR和AR都可能对这一目的有用，因为VR提供了一个一致的图形环境，使用户的交流更加清晰，而AR允许将与模拟机器人的交互置于真实的物理环境中，同时受益于图形增强。该项目开发了VR和AR格式的SVTE预曝光，以实现沉浸式用户培训，了解如何与机器人交流，并了解其功能和情感局限性。在整个SVTE交互过程中，系统将收集有关用户的信息，使机器人能够适应物理世界中的个性化交互。为了评估SVTE，在开发期间和完成后，该项目将进行一系列用户研究，首先是大学生，然后是高级生活设施中的老年人。在对老年用户进行的SVTE评估中，这项工作将侧重于使用机器人协助社会参与和促进防止社会孤立，这已被证明会提高发病率和死亡率。总体而言，该项目将包括开发一个混合现实开源测试平台，能够通过ROS和机器人的虚拟通信策略与物理机器人进行通信。使用来自这些虚拟模态的数据，这项工作将生成在物理世界中通常难以感知和创建的多模态用户模型。反过来，这些模型将允许非专家了解如何有效地与物理机器人互动。本研究将提供一个框架，用于评估多模态用户模型，以便在各种真实世界环境中进行自然交流、适应和个性化人机交互。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Robots designed for social interactions can enrich the quality of life for individuals and society in a myriad of ways, such as automating undesirable physical work, supporting activities of daily living, and facilitating social connections. However, seamless integration of robots into society depends on both people and robots understanding how to communicate naturally and effectively with each other. Humans need to understand the capabilities of a robot and develop trust in the robot. Robots need to understand the capabilities and interests of the human and adapt their behavior accordingly. This project will use virtual reality and augmented reality technologies to help people and robots to better understand each other before they meet in person. During a Shared Virtual Teaching Experience (SVTE), human users will be able to interact with a virtual version of a robot using virtual reality and augmented reality technologies. The SVTE will help both the robot and the users learn how to communicate with each other, build trust and rapport by sharing information about themselves, and adapt based on what each learns from the other. While an SVTE is a general approach for improving human-robot interactions, this project will focus on the needs of older adults, who may be hesitant to use and trust new technologies, including robots. Research has shown that older adults can benefit from social interactions with robots, especially when the robots help to create social connections with other older adults. This project helps to develop new approaches, leveraging virtual reality and augmented reality, to help users and robots work together. It will address societal needs of an aging population using socially assistive robots and advance the state of the art in human-robot interaction. It will also involve K-12 students in order to stimulate interest in science and engineering while engaging with the elderly community in the context of a real-world need for human-centered technology.This project uses immersive communication modalities of virtual reality (VR) and augmented reality (AR) to overcome the barriers to social communication that hinder seamless integration of co-robots into human everyday lives. Specifically, the project is developing an immersive mixed reality experience, Shared Virtual Teaching Experience (SVTE), allows human users to get accustomed to robots, and vice versa, before interacting together in the real world. Both VR and AR are potentially useful for this purpose, since VR provides a consistent graphical environment that makes communication clear for users, while AR allows for situating the interaction with a simulated robot in a real, physical environment, while benefiting from graphical enhancements. This project develops SVTE pre-exposures in both VR and AR formats to enable immersive user training on how to communicate with the robot and to understand its functional and affective limitations. Throughout SVTE interactions, the system will collect information about users that will allow the robot to adapt for personalized interactions in the physical world. To evaluate the SVTE, during development and when completed, this project will perform a series of user studies, first with university students and then with older adults in a senior living facility. In the SVTE evaluation with elderly users, this work will focus on the use of robots for assisting in social engagement and facilitation for preventing social isolation, which has been shown to raise morbidity and mortality. Overall, this project will include the development of a mixed reality open-source testbed capable of communicating with physical robots via ROS and virtual communication strategies for robots. Using data from those virtual modalities, this work will generate multimodal user models that are typically difficult to perceive and create within the physical world. The models will, in turn, allow non-experts to understand how to effectively interact with physical robots. This research will provide a framework for evaluating multimodal user models for naturally communicating, adapting, and personalizing human-robot interactions in various real world contexts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.