Accelerating Skill Acquisition in Complex Psychomotor Tasks via an Intelligent Extended Reality Tutoring System

制造业、医学实验室、建筑业和许多其他工作要求工人学习复杂的生理“精神运动”任务，这些任务结合了感知和运动技能。这些课程通常在实际工作现场采用学徒模式进行教学，这既提高了生产率，也增加了工人的安全风险。此外，对于如何评估受训者在这些任务中的技能水平，以及如何根据这些评估调整培训做法，所知相对较少。该项目通过开发新一代智能辅导系统来解决这些问题，该系统结合了扩展现实（XR）、人工智能（AI）和物联网（IoT）技术，以支持现代高度自动化制造设施所需的复杂技能的培训和评估。高水平的想法是，由XR耳机、可穿戴设备、摄像头和物联网传感器捕获的新数据源可用于建立精神运动技能发展模型，以及提供个性化、及时的指导指导的新方法。通过与制造业咨询公司、当地社区学院和K-12学校的合作，该项目将提高不同学习者和专业人员的技能发展，并扩大对先进制造业职业的兴趣。项目团队汇集了工程、认知心理学、学习科学、游戏设计和XR方面的专业知识，为制造业工人学习新技能提供即时、个性化、情境感知的学习框架，为学习科学和学习技术做出了根本性贡献。在学习方面，项目团队将检查特定精神运动任务的专业知识发展阶段，以及适应性干预对学习者参与度、表现增益和准确性的有效性。先进制造场景中的虚拟现实（VR）游戏将用于收集生态有效的基线数据，并为更多新手学习者准备现实世界的任务表现。在技术方面，项目团队将建立并验证一个智能XR辅导系统，以加速学习由任务结构和人类信息处理需求引起的高度复杂的精神运动任务。该技术的创新方面包括数据驱动的活动理解（例如，任务步骤识别和错误检测）和用户建模（例如，认知负载检测），通过新颖的多模态人工智能架构，旨在处理和融合从增强现实（AR）耳机、捕获生理数据的可穿戴设备、摄像头、物联网传感器和制造机器捕获的数据。学习和技术创新都将通过广泛的实验室研究得到验证；总之，这项工作将导致一种智能反馈算法，可以根据学习者的专业知识动态地调整反馈的性质、频率和深度，以促进最佳学习和提高能力的速度。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Manufacturing, medical laboratory, construction, and many other jobs require workers to learn complex physical “psychomotor” tasks that combine both perceptual and motor skills. These are often taught using an apprenticeship model on real jobsites, which raises both productivity and safety risks for workers. Further, relatively little is known about how to assess trainees’ skill levels in these tasks and to adapt training practices based on those assessments. This project tackles these problems by developing a new generation of intelligent tutoring systems that combine extended reality (XR), artificial intelligence (AI) and Internet-of-things (IoT) technologies to support training and assessment of complex skills required by modern, highly automated manufacturing facilities. The high level idea is that new sources of data captured by XR headsets, wearable devices, cameras, and IoT sensors can be used to build models of psychomotor skill development and new methods for providing personalized, just-in-time coaching guidance. Through partnerships with manufacturing consulting firms, local community colleges, and K-12 schools, the project will enhance the skill development of a diverse population of learners and professionals and expand interest in advanced manufacturing careers. The project team brings together expertise in engineering, cognitive psychology, learning sciences, game design, and XR, to make fundamental contributions to both learning science and learning technologies around just-in-time, personalized, context-aware provision of learning scaffolds for manufacturing workers learning new skills. On the learning side, the project team will examine the stages of expertise development for specific psychomotor tasks, and the effectiveness of adaptive interventions on learners’ engagement, performance gains, and accuracy. A virtual reality (VR) game in an advanced manufacturing scenario will be used to collect ecologically valid baseline data and prepare more novice learners for real-world task performance. On the technology side, the project team will build and validate an intelligent XR tutoring system to accelerate the learning of psychomotor tasks with high complexity that arises from task structures and human information processing requirements. The innovative aspects of the technology include data-driven activity understanding (e.g., task step identification and error detection) and user modeling (e.g., cognitive load detection), through novel multimodal AI architectures designed to process and fuse data captured from augmented reality (AR) headsets, wearables that capture physiological data, cameras, IoT sensors, and manufacturing machines. Both learning and technology innovations will be validated through extensive laboratory studies; together, the work will lead to an intelligent feedback algorithm to dynamically adapt the nature, frequency, and depth of feedback to the expertise of the learner to facilitate optimal learning and speed-to-competence.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.