Accelerating Skill Acquisition in Complex Psychomotor Tasks via an Intelligent Extended Reality Tutoring System

制造业、医学实验室、建筑业和许多其他工作需要工人学习复杂的物理“心理”任务，这些任务联合收割机感知和运动技能。这些通常是在真实的工作场所使用学徒模式教授的，这提高了工人的生产力和安全风险。此外，对于如何评估受训人员在这些任务中的技能水平以及如何根据这些评估调整培训做法，人们知之甚少。该项目通过开发新一代智能辅导系统来解决这些问题，该系统结合了联合收割机延展实境（XR），人工智能（AI）和物联网（IoT）技术，以支持现代化，高度自动化的制造设施所需的复杂技能的培训和评估。高层次的想法是，XR耳机、可穿戴设备、摄像头和物联网传感器捕获的新数据源可用于构建心理技能发展模型，以及提供个性化、即时辅导指导的新方法。通过与制造业咨询公司，当地社区学院和K-12学校的合作，该项目将加强学习者和专业人士的多样化人口的技能发展，并扩大对先进制造业职业的兴趣。该项目团队汇集了工程、认知心理学、学习科学、游戏设计和XR方面的专业知识，为学习科学和学习技术做出了根本性的贡献，为制造业工人学习新技能提供了及时、个性化、情境感知的学习支架。在学习方面，项目团队将研究特定心理任务的专业知识发展阶段，以及适应性干预措施对学习者参与，绩效收益和准确性的有效性。先进制造场景中的虚拟现实（VR）游戏将用于收集生态有效的基线数据，并为现实世界的任务性能准备更多的新手学习者。在技术方面，项目团队将构建并验证一个智能XR辅导系统，以加速学习由任务结构和人类信息处理要求引起的高复杂性心理任务。该技术的创新方面包括数据驱动的活动理解（例如，任务步骤识别和错误检测）和用户建模（例如，认知负荷检测），通过新颖的多模态AI架构，旨在处理和融合从增强现实（AR）耳机、捕获生理数据的可穿戴设备、相机、物联网传感器和制造机器捕获的数据。将通过广泛的实验室研究验证学习和技术创新;这项工作将导致智能反馈算法，以动态地适应自然，频率，和深度的反馈，以促进最佳的学习和速度，该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准。

Manufacturing, medical laboratory, construction, and many other jobs require workers to learn complex physical “psychomotor” tasks that combine both perceptual and motor skills. These are often taught using an apprenticeship model on real jobsites, which raises both productivity and safety risks for workers. Further, relatively little is known about how to assess trainees’ skill levels in these tasks and to adapt training practices based on those assessments. This project tackles these problems by developing a new generation of intelligent tutoring systems that combine extended reality (XR), artificial intelligence (AI) and Internet-of-things (IoT) technologies to support training and assessment of complex skills required by modern, highly automated manufacturing facilities. The high level idea is that new sources of data captured by XR headsets, wearable devices, cameras, and IoT sensors can be used to build models of psychomotor skill development and new methods for providing personalized, just-in-time coaching guidance. Through partnerships with manufacturing consulting firms, local community colleges, and K-12 schools, the project will enhance the skill development of a diverse population of learners and professionals and expand interest in advanced manufacturing careers. The project team brings together expertise in engineering, cognitive psychology, learning sciences, game design, and XR, to make fundamental contributions to both learning science and learning technologies around just-in-time, personalized, context-aware provision of learning scaffolds for manufacturing workers learning new skills. On the learning side, the project team will examine the stages of expertise development for specific psychomotor tasks, and the effectiveness of adaptive interventions on learners’ engagement, performance gains, and accuracy. A virtual reality (VR) game in an advanced manufacturing scenario will be used to collect ecologically valid baseline data and prepare more novice learners for real-world task performance. On the technology side, the project team will build and validate an intelligent XR tutoring system to accelerate the learning of psychomotor tasks with high complexity that arises from task structures and human information processing requirements. The innovative aspects of the technology include data-driven activity understanding (e.g., task step identification and error detection) and user modeling (e.g., cognitive load detection), through novel multimodal AI architectures designed to process and fuse data captured from augmented reality (AR) headsets, wearables that capture physiological data, cameras, IoT sensors, and manufacturing machines. Both learning and technology innovations will be validated through extensive laboratory studies; together, the work will lead to an intelligent feedback algorithm to dynamically adapt the nature, frequency, and depth of feedback to the expertise of the learner to facilitate optimal learning and speed-to-competence.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.