Signing Avatars & Immersive Learning (SAIL): Development and Testing of a Novel Embodied Learning Environment

学习美国手语（ASL）的改进资源需求量很大。ASL的传统教育材料往往包括书籍和视频，但在使用尖端技术来利用ASL的视觉空间性质以改善学习成果方面进展有限。交互式口语头像已经成为口语教学的宝贵学习工具，而签名头像的潜在用途尚未得到充分探讨。这项EArly探索性研究资助的目的是调查一个系统的可行性，在这个系统中，签名化身（由动作捕捉记录构建的计算机动画虚拟人）在沉浸式虚拟环境中教授用户ASL。该系统被称为签名化身沉浸式学习（SAIL）。该项目的重点是开发和测试这种全新的ASL学习工具，促进在STEM中纳入代表性不足的少数民族。这项工作有可能大大推进虚拟现实，美国手语教学和体现learning.This项目的领域利用认知神经科学的体现learning.This测试的SAIL系统。最终的目标是开发一个原型的系统，并测试其使用的听力非签名者的样本。签名头像是使用手语中本地聋人签名者的动作捕捉记录创建的。这些化身被放置在通过头戴式护目镜访问的虚拟现实场景中。用户进入虚拟现实环境，用户自己的动作通过手势跟踪系统捕获。一个“教师”化身指导用户通过一个互动的美国手语课程，涉及观察和生产的迹象。用户学习美国手语的迹象，从第一人称的角度和第三人称的角度。第一人称视角的纳入可能会增强体现学习过程的潜力。随着SAIL的发展，该项目涉及进行脑电图（EEG）实验，以检查大脑的感觉运动系统如何参与SAIL提供的体验。在观察另一个人的手势过程中，感觉运动皮层的神经活动程度提供了观察者如何在SAIL中处理手势的洞察力。该项目团队率先整合了多种技术：化身、动作捕捉系统、虚拟现实、手势跟踪和EEG，目标是在改进手语学习工具方面取得进展。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Improved resources for learning American Sign Language (ASL) are in high demand. Traditional educational materials for ASL tend to include books and videos, but there has been limited progress in using cutting-edge technologies to harness the visual-spatial nature of ASL for improved learning outcomes. Interactive speaking avatars have become valuable learning tools for spoken language instruction, whereas the potential uses of signing avatars have not been adequately explored. The aim of this EArly Grant for Exploratory Research is to investigate the feasibility of a system in which signing avatars (computer-animated virtual humans built from motion capture recordings) teach users ASL in an immersive virtual environment. The system is called Signing Avatars & Immersive Learning (SAIL). The project focuses on developing and testing this entirely novel ASL learning tool, fostering the inclusion of underrepresented minorities in STEM. This work has the potential to substantially advance the fields of virtual reality, ASL instruction, and embodied learning.This project leverages the cognitive neuroscience of embodied learning to test the SAIL system. The ultimate goal is to develop a prototype of the system and test its use in a sample of hearing non-signers. Signing avatars are created using motion capture recordings of native deaf signers signing in ASL. The avatars are placed in a virtual reality landscape accessed via head-mounted goggles. Users enter the virtual reality environment, and the user's own movements are captured via a gesture-tracking system. A "teacher" avatar guides users through an interactive ASL lesson involving both the observation and production of signs. Users learn ASL signs from both the first-person perspective and the third-person perspective. The inclusion of the first-person perspective may enhance the potential for embodied learning processes. Following the development of SAIL, the project involves conducting an electroencephalography (EEG) experiment to examine how the sensorimotor systems of the brain are engaged by the embodied experiences provided in SAIL. The extent of neural activity in the sensorimotor cortex during viewing of another person signing provides insight into how the observer is processing the signs within SAIL. The project team pioneers the integration of multiple technologies: avatars, motion capture systems, virtual reality, gesture tracking, and EEG with the goal of making progress toward an improved tool for sign language learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.