NRI: FND: Communicate, Share, Adapt: A Mixed Reality Framework for Facilitating Robot Integration and Customization

NRI：FND：沟通、共享、适应：促进机器人集成和定制的混合现实框架

• 批准号: 1925083
• 负责人: Maja Matarić
• 金额: $ 75万
• 依托单位: University of Southern California
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1925083&HistoricalAwards=false
• 关键词: NRI; FND; Communicate; Share; Adapt

Robots designed for social interactions can enrich the quality of life for individuals and society in a myriad of ways, such as automating undesirable physical work, supporting activities of daily living, and facilitating social connections. However, seamless integration of robots into society depends on both people and robots understanding how to communicate naturally and effectively with each other. Humans need to understand the capabilities of a robot and develop trust in the robot. Robots need to understand the capabilities and interests of the human and adapt their behavior accordingly. This project will use virtual reality and augmented reality technologies to help people and robots to better understand each other before they meet in person. During a Shared Virtual Teaching Experience (SVTE), human users will be able to interact with a virtual version of a robot using virtual reality and augmented reality technologies. The SVTE will help both the robot and the users learn how to communicate with each other, build trust and rapport by sharing information about themselves, and adapt based on what each learns from the other. While an SVTE is a general approach for improving human-robot interactions, this project will focus on the needs of older adults, who may be hesitant to use and trust new technologies, including robots. Research has shown that older adults can benefit from social interactions with robots, especially when the robots help to create social connections with other older adults. This project helps to develop new approaches, leveraging virtual reality and augmented reality, to help users and robots work together. It will address societal needs of an aging population using socially assistive robots and advance the state of the art in human-robot interaction. It will also involve K-12 students in order to stimulate interest in science and engineering while engaging with the elderly community in the context of a real-world need for human-centered technology.This project uses immersive communication modalities of virtual reality (VR) and augmented reality (AR) to overcome the barriers to social communication that hinder seamless integration of co-robots into human everyday lives. Specifically, the project is developing an immersive mixed reality experience, Shared Virtual Teaching Experience (SVTE), allows human users to get accustomed to robots, and vice versa, before interacting together in the real world. Both VR and AR are potentially useful for this purpose, since VR provides a consistent graphical environment that makes communication clear for users, while AR allows for situating the interaction with a simulated robot in a real, physical environment, while benefiting from graphical enhancements. This project develops SVTE pre-exposures in both VR and AR formats to enable immersive user training on how to communicate with the robot and to understand its functional and affective limitations. Throughout SVTE interactions, the system will collect information about users that will allow the robot to adapt for personalized interactions in the physical world. To evaluate the SVTE, during development and when completed, this project will perform a series of user studies, first with university students and then with older adults in a senior living facility. In the SVTE evaluation with elderly users, this work will focus on the use of robots for assisting in social engagement and facilitation for preventing social isolation, which has been shown to raise morbidity and mortality. Overall, this project will include the development of a mixed reality open-source testbed capable of communicating with physical robots via ROS and virtual communication strategies for robots. Using data from those virtual modalities, this work will generate multimodal user models that are typically difficult to perceive and create within the physical world. The models will, in turn, allow non-experts to understand how to effectively interact with physical robots. This research will provide a framework for evaluating multimodal user models for naturally communicating, adapting, and personalizing human-robot interactions in various real world contexts.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

为社交而设计的机器人可以通过各种方式丰富个人和社会的生活质量，例如自动化不受欢迎的体力工作，支持日常生活活动，以及促进社会联系。然而，机器人能否无缝融入社会，有赖于人和机器人都懂得如何自然有效地相互沟通。人类需要了解机器人的能力，并建立对机器人的信任。机器人需要理解人类的能力和兴趣，并相应地调整他们的行为。该项目将使用虚拟现实和增强现实技术，帮助人们和机器人在面对面之前更好地了解对方。在共享虚拟教学体验(SVTE)期间，人类用户将能够使用虚拟现实和增强现实技术与虚拟版本的机器人互动。SVTE将帮助机器人和用户学习如何相互通信，通过分享关于自己的信息来建立信任和融洽，并根据各自从对方那里学到的东西进行调整。虽然SVTE是改善人与机器人交互的一般方法，但该项目将专注于老年人的需求，他们可能对使用和信任包括机器人在内的新技术犹豫不决。研究表明，老年人可以从与机器人的社交互动中受益，特别是当机器人帮助建立与其他老年人的社交联系时。该项目帮助开发利用虚拟现实和增强现实的新方法，以帮助用户和机器人协同工作。它将使用社交辅助机器人解决老龄化人口的社会需求，并促进人与机器人交互的最新水平。该项目还将让K-12年级的学生参与，以激发他们对科学和工程的兴趣，同时在现实世界对以人为中心的技术的需求背景下与老年社区互动。该项目使用虚拟现实(VR)和增强现实(AR)的沉浸式通信模式，以克服社交沟通的障碍，这些障碍阻碍了协作机器人无缝融入人类的日常生活。具体地说，该项目正在开发一种身临其境的混合现实体验，即共享虚拟教学体验(SVTE)，允许人类用户在现实世界中一起互动之前习惯机器人，反之亦然。VR和AR对于这一目的都是潜在的有用的，因为VR提供了一致的图形环境，使用户能够清楚地进行交流，而AR允许在真实的物理环境中定位与模拟机器人的交互，同时受益于图形增强。该项目开发VR和AR两种格式的SVTE预曝光，以使用户能够身临其境地培训如何与机器人沟通，并了解其功能和情感限制。在整个SVTE交互过程中，系统将收集有关用户的信息，使机器人能够适应物理世界中的个性化交互。为了评估SVTE，在开发过程中和完成后，该项目将进行一系列用户研究，首先是大学生，然后是老年生活设施中的老年人。在对老年用户进行的SVTE评估中，这项工作将侧重于使用机器人协助社会参与和促进防止社会孤立，这已被证明会增加发病率和死亡率。总体而言，该项目将包括开发一个混合现实开源试验台，能够通过ROS和机器人的虚拟通信策略与实体机器人进行通信。使用来自这些虚拟模式的数据，这项工作将生成通常难以在物理世界中感知和创建的多模式用户模型。反过来，这些模型将允许非专家了解如何有效地与物理机器人交互。这项研究将为评估在各种现实世界环境中自然交流、适应和个性化人-机器人交互的多模式用户模型提供一个框架。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

MoveToCode: An Embodied Augmented Reality Visual Programming Language with an Autonomous Robot Tutor for Promoting Student Programming Curiosity

MoveToCode：一种带有自主机器人导师的具体增强现实可视化编程语言，可提高学生的编程好奇心

• 发表时间: 2023
• 期刊: IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN 2023
• 作者: Thomas R. Groechel, İpek Göktan