New Dimensions of ASL Learning: Implementing and Testing Signing Avatars & Immersive Learning (SAIL 2)

美国手语(ASL)的高度空间三维特性为技术支持的ASL教学制造了严重的障碍。如果ASL学习者可以通过基于虚拟现实的类似游戏的环境从母语手语教师那里获得高质量的ASL教学，情况会怎样？这个项目是从NSF资助的签名化身和沉浸式学习(SAIL)项目的先前工作开始的。SAIL项目产生了一个在虚拟现实中沉浸式手语学习环境的工作原型。目前的项目经过了原型阶段的扩展，进入了一个全面的ASL学习体验。在名为SAIL 2的新版SAIL中，研究团队正在开发一个更完整的系统，用户可以进入虚拟现实，并与手势化身(根据动作捕捉记录构建的计算机动画虚拟人)互动，后者教用户ASL词汇。获得手语是许多聋人健康发展的关键，但在受时间和资源限制的情况下，获得高质量的美国手语教学仍然是一个重大挑战。SAIL 2为更多地学习ASL奠定了基础，这有可能改善聋儿和成年人的生活。该项目的重点是开发和测试这一全新的高级语言学习工具，并促进将代表性不足的少数群体纳入STEM。这项工作有可能极大地推动虚拟现实、ASL教学和具体化学习领域的发展。沉浸式虚拟现实特别适合于高度空间的手语。SAIL 2项目利用头盔虚拟现实和高质量的签名化身来创建游戏化的ASL学习系统。SAIL 2将是虚拟现实中唯一不需要用户佩戴专门的手套或其他外围设备的ASL学习系统。该项目开发了综合SAIL 2系统的功能版本，在设计过程中进行的用户测试指导了开发的细节。该系统的主要功能包括通过手部跟踪摄像头进行手势识别、纠正反馈和游戏化体验。在SAIL 2的设计和开发之后，研究小组进行了行为研究来评估SAIL 2的学习结果。具体的学习结果的评估包括对ASL词汇的理解和手势产生的准确性。由于手语的体验性，学习的神经基础的机械测量，包括感觉运动皮质的参与，是通过脑电(EEG)获得的。神经振荡活动的模式提供了与使用SAIL 2相关的大脑活动短期变化的洞察。认知神经科学实验建立在先前确定支持手语感知的神经过程的研究基础上，总体上，该项目扩展了高保真运动捕捉记录、虚拟形象创建和虚拟现实方面的技术进步。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The highly-spatial three-dimensional nature of American Sign Language (ASL) has created a serious barrier to technology-supported ASL instruction. What if ASL learners could access high-quality ASL instruction from native sign language instructors through a virtual reality-based, game-like environment? This project launches from prior work on the NSF-funded Signing Avatars & Immersive Learning (SAIL) project. The SAIL project yielded a working prototype of an immersive sign language learning environment in virtual reality. The current project expands past the prototype stage into a fully-fledged ASL learning experience. In the new version of SAIL, called SAIL 2, the research team is developing a more complete system where users enter virtual reality and interact with signing avatars (computer-animated virtual humans built from motion capture recordings) who teach users ASL vocabulary. Access to signed language is key to healthy development for many deaf individuals, but it remains a major challenge when access to high quality ASL instruction in limited by time and resources. SAIL 2 sets a foundation for greater access to learning ASL, which has potential for improving the lives of deaf children and adults. The project focuses on developing and testing this entirely novel ASL learning tool and fostering the inclusion of underrepresented minorities in STEM. This work has the potential to substantially advance the fields of virtual reality, ASL instruction, and embodied learning.Immersive virtual reality is particularly well suited for highly spatial signed languages. The SAIL 2 project leverages head-mounted virtual reality and high-quality signing avatars to create a gamified ASL-learning system. SAIL 2 will be the only ASL learning system in virtual reality which does not require the user to wear specialized gloves or other peripheral devices. The project develops a functioning version of the comprehensive SAIL 2 system, and user testing during the design process guides the details of development. Key features of the system include sign recognition through hand tracking cameras, corrective feedback, and a gamified experience. Following the design and development of SAIL 2, the research team conducts behavioral research to evaluate the learning outcomes of SAIL 2. Evaluation of specific learning outcomes includes both understanding of ASL vocabulary and accuracy of sign production. Because of the embodied nature of signed language, mechanistic measures of the neural substrates of learning, including engagement of the sensorimotor cortices, are obtained through electroencephalography (EEG). The patterns of neural oscillatory activity provide insight into short-term changes in brain activity associated with using SAIL 2. The cognitive neuroscience experiment builds on previous research identifying the neural processes supporting sign language perception, and overall this project extends technological advances in high-fidelity motion capture recordings, avatar creation, and virtual reality.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.