New Dimensions of ASL Learning: Implementing and Testing Signing Avatars & Immersive Learning (SAIL 2)

美国手语的高度空间化、立体化特征给手语教学带来了严重的技术障碍。如果美国手语学习者可以通过基于虚拟现实的游戏环境从本地手语教师那里获得高质量的美国手语教学，会怎么样？该项目是从NSF资助的Signing Avatars Immersive Learning（SAIL）项目的前期工作开始的。SAIL项目在虚拟现实中产生了沉浸式手语学习环境的工作原型。目前的项目扩展到过去的原型阶段，成为一个完全成熟的ASL学习经验。在新版本的SAIL 2中，研究团队正在开发一个更完整的系统，用户可以进入虚拟现实并与签名化身（由动作捕捉记录构建的计算机动画虚拟人）进行交互，后者教授用户ASL词汇。获得手语是许多聋人健康发展的关键，但在时间和资源有限的情况下获得高质量的ASL教学仍然是一个重大挑战。SAIL 2为更多地学习美国手语奠定了基础，这有可能改善聋哑儿童和成年人的生活。该项目的重点是开发和测试这种全新的ASL学习工具，并促进将代表性不足的少数民族纳入STEM。这项工作有可能大大推进虚拟现实，美国手语教学和体现learning.Immersive虚拟现实领域特别适合高度空间签名语言。SAIL 2项目利用头戴式虚拟现实和高质量的签名头像来创建一个游戏化的ASL学习系统。SAIL 2将是虚拟现实中唯一的ASL学习系统，它不需要用户戴专门的手套或其他外围设备。该项目开发了一个全面的SAIL 2系统的功能版本，设计过程中的用户测试指导了开发的细节。该系统的主要功能包括通过手部跟踪摄像头进行手势识别，纠正反馈和游戏化体验。在设计和开发SAIL 2之后，研究团队进行了行为研究，以评估SAIL 2的学习成果。具体的学习成果的评估包括对美国手语词汇的理解和手势产生的准确性。由于手语的具体性质，学习的神经基质的机械措施，包括参与的感觉运动皮层，通过脑电图（EEG）获得。神经振荡活动的模式提供了与使用SAIL 2相关的大脑活动短期变化的见解。认知神经科学实验建立在以前的研究基础上，确定了支持手语感知的神经过程，总体而言，该项目扩展了高保真动作捕捉记录，化身创作和虚拟现实的技术进步。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

The highly-spatial three-dimensional nature of American Sign Language (ASL) has created a serious barrier to technology-supported ASL instruction. What if ASL learners could access high-quality ASL instruction from native sign language instructors through a virtual reality-based, game-like environment? This project launches from prior work on the NSF-funded Signing Avatars & Immersive Learning (SAIL) project. The SAIL project yielded a working prototype of an immersive sign language learning environment in virtual reality. The current project expands past the prototype stage into a fully-fledged ASL learning experience. In the new version of SAIL, called SAIL 2, the research team is developing a more complete system where users enter virtual reality and interact with signing avatars (computer-animated virtual humans built from motion capture recordings) who teach users ASL vocabulary. Access to signed language is key to healthy development for many deaf individuals, but it remains a major challenge when access to high quality ASL instruction in limited by time and resources. SAIL 2 sets a foundation for greater access to learning ASL, which has potential for improving the lives of deaf children and adults. The project focuses on developing and testing this entirely novel ASL learning tool and fostering the inclusion of underrepresented minorities in STEM. This work has the potential to substantially advance the fields of virtual reality, ASL instruction, and embodied learning.Immersive virtual reality is particularly well suited for highly spatial signed languages. The SAIL 2 project leverages head-mounted virtual reality and high-quality signing avatars to create a gamified ASL-learning system. SAIL 2 will be the only ASL learning system in virtual reality which does not require the user to wear specialized gloves or other peripheral devices. The project develops a functioning version of the comprehensive SAIL 2 system, and user testing during the design process guides the details of development. Key features of the system include sign recognition through hand tracking cameras, corrective feedback, and a gamified experience. Following the design and development of SAIL 2, the research team conducts behavioral research to evaluate the learning outcomes of SAIL 2. Evaluation of specific learning outcomes includes both understanding of ASL vocabulary and accuracy of sign production. Because of the embodied nature of signed language, mechanistic measures of the neural substrates of learning, including engagement of the sensorimotor cortices, are obtained through electroencephalography (EEG). The patterns of neural oscillatory activity provide insight into short-term changes in brain activity associated with using SAIL 2. The cognitive neuroscience experiment builds on previous research identifying the neural processes supporting sign language perception, and overall this project extends technological advances in high-fidelity motion capture recordings, avatar creation, and virtual reality.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.