Teaching American Sign Language in Mixed Reality
Teaching American Sign Language in Mixed Reality
复制标题
DOI：
10.1145/3432211
复制
发表时间：
2020-12
影响因子：
--
通讯作者：
Qijia Shao;A. Sniffen;Julien Blanchet;Megan E. Hillis;Xinyu Shi;Themistoklis K. Haris;Jason Liu;Jason Lamberton;Melissa Malzkuhn;Lorna C. Quandt;James Mahoney;David J. M. Kraemer;Xia Zhou;Devin J. Balkcom
中科院分区：
文献类型：
--
作者：
Qijia Shao;A. Sniffen;Julien Blanchet;Megan E. Hillis;Xinyu Shi;Themistoklis K. Haris;Jason Liu;Jason Lamberton;Melissa Malzkuhn;Lorna C. Quandt;James Mahoney;David J. M. Kraemer;Xia Zhou;Devin J. Balkcom
本文提出了一个整体系统，以扩大美国手语（ASL）的词汇教学和学习。该系统利用最新的混合现实技术，允许用户在沉浸式学习环境中感知自己的双手，并以第一人称和第三人称视角进行动作演示和练习。精确的运动传感用于记录和评估运动，提供针对特定学习者的实时反馈。作为评估的一部分，学习者的动作匹配的功能来自汉堡符号系统（HNS）开发的手语语言学家。我们开发了一个原型来评估基于混合现实的交互式运动教学的效果。60名参与者的结果显示，与传统的基于桌面的非交互式学习相比，使用我们的系统学习ASL标志有统计学上的显着改善。我们希望这种方法最终能够允许数千个标志的教学和指导实践。
This paper presents a holistic system to scale up the teaching and learning of vocabulary words of American Sign Language (ASL). The system leverages the most recent mixed-reality technology to allow the user to perceive her own hands in an immersive learning environment with first- and third-person views for motion demonstration and practice. Precise motion sensing is used to record and evaluate motion, providing real-time feedback tailored to the specific learner. As part of this evaluation, learner motions are matched to features derived from the Hamburg Notation System (HNS) developed by sign-language linguists. We develop a prototype to evaluate the efficacy of mixed-reality-based interactive motion teaching. Results with 60 participants show a statistically significant improvement in learning ASL signs when using our system, in comparison to traditional desktop-based, non-interactive learning. We expect this approach to ultimately allow teaching and guided practice of thousands of signs.