CAREER: Extended Reality Meets AI: Designing Interactions for Novel Human-AI Systems

人机交互研究的目的是开发以用户为中心的系统，提供有效的终身学习，涵盖所有知识领域和技能。一个关键领域是学习具有巨大实用性（如手术训练）、科学性（如理解运动记忆）和文化性（如音乐和体育）兴趣的运动或身体技能。为了学习新的运动技能或在受伤后恢复它们，人们通常需要与可以指导，评估表现，提供反馈，支持和激励的专家进行强化训练或康复治疗。与专家合作可以确保学习者达到预期的效果，正确锻炼，避免受伤，并深入了解自己的表现。但是，专家并不总是可以联系到的，特别是在对专家的需求超过供应的情况下，或者当人们受到个人、金钱或地理限制而无法联系到专家的情况下。为了应对缺乏专家的情况，在家学习的替代方案包括书籍，视频，应用程序和在线服务（如ClassPass，Peloton，Mirror）。虽然这些选项成功地提供了解释和演示，但没有一个选项可以向受训者提供有意义的实时反馈。该项目利用人工智能（AI）和延展实境（XR）技术的最新进展来构建新型的交互式运动技能训练和康复系统，用户可以从以3D化身表示的AI代理接收实时反馈。通过构建模拟人类专家实时语言和非语言行为的人工智能代理，该项目将提供有关运动技能学习（特别是康复治疗）背景下管理人类-人工智能交互的基本原则的基础知识。该项目有可能通过以低成本、低干扰和大规模减少经济或心理障碍，扩大在家中获得运动技能学习和康复的机会。鉴于算法越来越多地进入社会领域，理解人类和人工智能主体如何互动对设计负责任和可解释的人工智能具有广泛的潜在影响。该项目的目标是研究用于运动技能学习和康复的AI-XR（人工智能-延展实境）系统的设计。具体来说，该项目提出了一种新的人机界面范例，其中AI在XR中表示为3D人形代理，可以模仿人类教练的实时语言和非语言行为，以在没有人类专家的情况下支持运动学习。人类专家训练师在提供必要的解释、演示和纠正身体运动表现时，通过视觉、听觉和触觉通道利用多模态反馈。为了模拟受训者和专家培训师之间的自然互动，人工智能代理将被设计为实时到近实时地识别和响应用户的身体动作，表现得像人类一样，并使用语言和非语言行为（如眼睛凝视，头部摆动，面部表情和手势）进行交流。因此，开发用于运动学习的人工智能代理提出了许多技术和可推广的研究挑战，包括（i）如何设计有效的多模态指令和反馈机制，以及（ii）如何实现用户和人工智能代理之间的自然交互。为了应对这些挑战，该项目将实施新的（i）用于全面身体运动和错误分析的算法和人工智能模型;（ii）多模式指令和反馈技术;以及（iii）展示新的人机交互范例的健身训练和康复应用程序。AI代理和系统设计将通过与用户进行对照实验进行评估。本研究项目的评估将通过三个应用程序进行：（1）体能，（2）运动损伤康复，（3）面瘫康复。本研究的部分实施在发挥和同龄人为基础的教育活动的初中和高中学生将被设计为互动和参与，同时引发解决问题和批判性思维。该奖项反映了NSF的法定使命，通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Research in human-computer interaction aims to develop user-centered systems that provide access to effective lifelong learning across the full range of knowledge domains and skills. A key area is learning motor or physical skills of tremendous practical (such as training for surgery), scientific (such as understanding motor memory) and cultural (such as music and sports) interest. To learn new motor skills or recover them after an injury, people often need intensive training or rehabilitation therapy with an expert who can instruct, assess performance, provide feedback, support, and motivate. Working with an expert can ensure learners achieve desired results, exercise correctly, avoid injury, and gain insights into their own performance. But an expert may not always be accessible or available, particularly in situations where demand for experts exceeds supply or when people have personal, monetary, or geographic limitations that prevent access. To deal with the absence of an expert, alternatives for learning at home include books, videos, apps, and online services (such as ClassPass, Peloton, Mirror). Although these options successfully provide explanations and demonstrations, none of them can provide meaningful real-time feedback to the trainees. This project leverages recent advances in Artificial Intelligence (AI) and eXtended Reality (XR) technologies to build new types of interactive motor skill training and rehabilitation systems where users receive real-time feedback from AI agents represented as 3D avatars. By building AI agents that emulate the real-time verbal and non-verbal behaviors of a human expert, the project will provide foundational knowledge on underlying principles that govern human-AI interactions in the context of motor skill learning (specifically, rehabilitation therapy). The project has the potential to broaden access to motor skill learning and rehabilitation at home by reducing financial or psychological barriers at low cost, with low intrusion, and at scale. Given that algorithms are increasingly making inroads into societally consequential domains, understanding how humans and AI agents interact has broad potential implications for the design of responsible and explainable AI. The goal of this project is to investigate the design of AI-XR (artificial intelligence - extended reality) systems for motor skill learning and rehab. Specifically, the project advances a new human-AI interface paradigm in which the AI is represented as a 3D humanoid agent in XR that can mimic the real-time verbal and non-verbal behaviors of a human trainer to support motor learning in the absence of a human expert. Human expert trainers make use of multimodal feedback through visual, auditory, and haptic channels when providing the necessary explanations, demonstrations, and corrections of physical motion performance. To emulate natural interactions between a trainee and an expert trainer, the AI agents will be designed to recognize and respond to a user’s physical actions in real-time to near-real-time, appear and act human-like, and use verbal and non-verbal behaviors (such as eye gaze, head nods, facial expressions, and gestures) to communicate. Developing AI agents for motor learning therefore presents numerous technical and generalizable research challenges, including (i) how to design effective multimodal instruction and feedback mechanisms and (ii) how to enable natural interactions between the user and an AI agent. To tackle these challenges, the project will implement new (i) algorithms and AI models for comprehensive body motion and error analysis; (ii) multimodal instruction and feedback techniques; and (iii) applications for fitness training and rehab that demonstrate the new human-AI interaction paradigm. The AI agent and system designs will be evaluated by performing controlled experiments with users. The evaluation of this research project will be through three applications for: (1) physical fitness, (2) sports injury rehabilitation, and (3) facial paralysis rehabilitation. Implementations of parts of this research in play- and peer-based educational activities for middle and high school students will be designed to be interactive and engaging, while eliciting problem solving and critical thinking. The goal is to guide the next generation to think deeply about AI and its implications on human behavior and by extension, on society.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.