Signing Avatars & Immersive Learning (SAIL): Development and Testing of a Novel Embodied Learning Environment

对学习美国手语(ASL)的改进资源的需求很高。传统的ASL教育材料往往包括书籍和视频，但在利用ASL的视觉空间本质来改善学习结果方面，利用尖端技术取得的进展有限。交互式语音化身已经成为口语教学的宝贵学习工具，而手语化身的潜在用途还没有得到充分的探索。这项早期探索性研究赠款的目的是调查一种系统的可行性，在该系统中，签名化身(根据运动捕捉记录构建的计算机动画虚拟人)在身临其境的虚拟环境中教授用户ASL。这个系统被称为签名化身和沉浸式学习(SAIL)。该项目的重点是开发和测试这一全新的高级语言学习工具，促进将代表性不足的少数群体纳入STEM。这项工作有可能实质性地推动虚拟现实、ASL教学和具身学习领域的发展。该项目利用具身学习的认知神经科学来测试SAIL系统。最终目标是开发该系统的原型，并在听力非签名者的样本中测试其使用情况。手语头像是使用在ASL中签名的本地聋人手语者的动作捕捉记录创建的。化身被放置在通过头盔护目镜进入的虚拟现实景观中。用户进入虚拟现实环境，通过手势跟踪系统捕捉用户自己的动作。一个“老师”的化身引导用户通过一个互动的ASL课程，包括观察和产生手势。用户从第一人称视角和第三人称视角学习ASL手语。纳入第一人称观点可能会增强具体化学习过程的潜力。随着SAIL的发展，该项目涉及进行脑电(EEG)实验，以检查大脑的感觉运动系统是如何与SAIL提供的具体经验相联系的。在观看另一个人的手势时，感觉运动皮质中神经活动的程度可以让我们深入了解观察者是如何处理帆内的手势的。该项目团队率先整合了多种技术：化身、动作捕捉系统、虚拟现实、手势跟踪和脑电波，目标是在改进手语学习工具方面取得进展。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Improved resources for learning American Sign Language (ASL) are in high demand. Traditional educational materials for ASL tend to include books and videos, but there has been limited progress in using cutting-edge technologies to harness the visual-spatial nature of ASL for improved learning outcomes. Interactive speaking avatars have become valuable learning tools for spoken language instruction, whereas the potential uses of signing avatars have not been adequately explored. The aim of this EArly Grant for Exploratory Research is to investigate the feasibility of a system in which signing avatars (computer-animated virtual humans built from motion capture recordings) teach users ASL in an immersive virtual environment. The system is called Signing Avatars & Immersive Learning (SAIL). The project focuses on developing and testing this entirely novel ASL learning tool, fostering the inclusion of underrepresented minorities in STEM. This work has the potential to substantially advance the fields of virtual reality, ASL instruction, and embodied learning.This project leverages the cognitive neuroscience of embodied learning to test the SAIL system. The ultimate goal is to develop a prototype of the system and test its use in a sample of hearing non-signers. Signing avatars are created using motion capture recordings of native deaf signers signing in ASL. The avatars are placed in a virtual reality landscape accessed via head-mounted goggles. Users enter the virtual reality environment, and the user's own movements are captured via a gesture-tracking system. A "teacher" avatar guides users through an interactive ASL lesson involving both the observation and production of signs. Users learn ASL signs from both the first-person perspective and the third-person perspective. The inclusion of the first-person perspective may enhance the potential for embodied learning processes. Following the development of SAIL, the project involves conducting an electroencephalography (EEG) experiment to examine how the sensorimotor systems of the brain are engaged by the embodied experiences provided in SAIL. The extent of neural activity in the sensorimotor cortex during viewing of another person signing provides insight into how the observer is processing the signs within SAIL. The project team pioneers the integration of multiple technologies: avatars, motion capture systems, virtual reality, gesture tracking, and EEG with the goal of making progress toward an improved tool for sign language learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.