CHS: Medium: Collaborative Research: Immediate Feedback to Support Learning American Sign Language through Multisensory Recognition

美国手语（英语：American Sign Language，缩写：ASL）是美国50万人的主要交流方式，是一种与英语不同的语言，通过手势、面部表情和身体动作传达。 研究表明，失聪父母的失聪儿童比听力正常父母的失聪儿童阅读能力更好，这主要是因为当孩子和父母都是失聪时，沟通更好。 然而，超过80%的失聪或重听儿童是由听力正常的父母所生。 对于家长、老师和其他人来说，如何快速地学习美国手语以支持孩子的视觉语言习得是一个挑战。 可以自动识别ASL签名的各个方面并向这些ASL学生提供即时反馈的技术将为他们提供时间灵活的方式来练习和提高他们的签名技能。 这个项目的目标，其中涉及一个跨学科的研究人员团队在三个学院内的纽约（CUNY）与专业知识，在计算机视觉，人机交互，聋人和重听教育，是发现最有效的底层技术，用户界面设计，和教学使用的互动工具，以提供这样的即时，关于ASL识别的大多数先前工作集中于识别所执行的简单符号的小集合，但是当前技术在具有不受限制的词汇的句子的连续符号上不够准确。 PI将开发从根本上推进ASL部分识别的技术，即识别ASL的语言/性能属性，而不必识别整个符号序列，并自动确定性能是否流畅或包含错误。 本研究将包括五个方面的内容：（1）基于美国手语语言学和教学法，识别出一组可观察到的表征美国手语流利性的属性;（2）通过多模态融合，发现自动检测美国手语流利性属性的新技术（面部表情，手势和身体姿势）和多感官信息（RGB和深度视频）;（3）收集和注释学生和本地签名者以不同流利程度表演的手语RGBD视频数据集;（4）开发一个互动的美国手语学习工具，为美国手语学生提供关于他们的手语是否流利的即时反馈;以及（5）评估新算法的鲁棒性和ASL学习工具的有效性，包括其教育效益。 这项工作将导致计算机视觉技术在人类行为感知方面的进步，对ASL视频用户界面设计的新理解，以及革命性和具有成本效益的教育工具，以帮助ASL学习者实现流畅性，使用识别技术在短期内是强大和准确的。 项目成果将包括不同流利程度的视频数据集，这对未来的ASL语言学家或教师、学习ASL的学生和计算机视觉研究人员都很有价值。

American Sign Language (ASL) is a primary means of communication for 500,000 people in the United States and a distinct language from English, conveyed through hands, facial expressions, and body movements. Studies indicate that deaf children of deaf parents read better than deaf children of hearing parents, mainly due to better communication when both children and parents are deaf. However, more than 80% of children who are deaf or hard of hearing are born to hearing parents. It is challenging for parents, teachers, and other people in the life of a deaf child to learn ASL rapidly enough to support the visual language acquisition of the child. Technology that can automatically recognize aspects of ASL signing and provide instant feedback to these students of ASL would give them a time-flexible way to practice and improve their signing skills. The goal of this project, which involves an interdisciplinary team of researchers at three colleges within the City University of New York (CUNY) with expertise in computer vision, human-computer interaction, and Deaf and Hard of Hearing education, is to discover the most effective underlying technologies, user-interface design, and pedagogical use for an interactive tool to provide such immediate, automatic feedback for students of ASL.Most prior work on ASL recognition has focused on identifying a small set of simple signs performed, but current technology is not sufficiently accurate on continuous signing of sentences with an unrestricted vocabulary. The PIs will develop technologies to fundamentally advance ASL partial recognition, that is to identify linguistic/performance attributes of ASL without necessarily identifying the entire sequence of signs, and automatically determine if a performance is fluent or contains errors. The research will include five thrusts: (1) based on ASL linguistics and pedagogy, to identify a set of observable attributes indicating ASL fluency; (2) to discover new technologies for automatic detection of the ASL fluency attributes through fusion of multimodality (facial expression, hand gesture, and body pose) and multisensory information (RGB and Depth videos); (3) to collect and annotate a dataset of RGBD videos of ASL, performed at varied levels of fluency, by students and native signers; (4) to develop an interactive ASL learning tool that provides ASL students immediate feedback about whether their signing is fluent or not; and (5) to evaluate the robustness of the new algorithms and the effectiveness of the ASL learning tool, including its educational benefits. The work will lead to advances in computer vision technologies for human behavior perception, to new understanding of user-interface design with ASL video, and to a revolutionary and cost-effective educational tool to assist ASL learners achieve fluency, using recognition technologies that are robust and accurate in the near-term. Project outcomes will include a dataset of videos at varied fluency levels, which will be valuable for future ASL linguists or instructors, students learning ASL, and computer vision researchers.