CHS: Medium: Collaborative Research: Immediate Feedback to Support Learning American Sign Language through Multisensory Recognition

CHS：媒介：协作研究：通过多感官识别支持学习美国手语的即时反馈

• 批准号: 1400906
• 负责人: Matt Huenerfauth
• 金额: $ 53.8万
• 依托单位: CUNY Queens College
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-09-01 至 2014-10-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1400906&HistoricalAwards=false
• 关键词: CHS; Medium; Collaborative; Research; Immediate

American Sign Language (ASL) is a primary means of communication for 500,000 people in the United States and a distinct language from English, conveyed through hands, facial expressions, and body movements. Studies indicate that deaf children of deaf parents read better than deaf children of hearing parents, mainly due to better communication when both children and parents are deaf. However, more than 80% of children who are deaf or hard of hearing are born to hearing parents. It is challenging for parents, teachers, and other people in the life of a deaf child to learn ASL rapidly enough to support the visual language acquisition of the child. Technology that can automatically recognize aspects of ASL signing and provide instant feedback to these students of ASL would give them a time-flexible way to practice and improve their signing skills. The goal of this project, which involves an interdisciplinary team of researchers at three colleges within the City University of New York (CUNY) with expertise in computer vision, human-computer interaction, and Deaf and Hard of Hearing education, is to discover the most effective underlying technologies, user-interface design, and pedagogical use for an interactive tool to provide such immediate, automatic feedback for students of ASL.Most prior work on ASL recognition has focused on identifying a small set of simple signs performed, but current technology is not sufficiently accurate on continuous signing of sentences with an unrestricted vocabulary. The PIs will develop technologies to fundamentally advance ASL partial recognition, that is to identify linguistic/performance attributes of ASL without necessarily identifying the entire sequence of signs, and automatically determine if a performance is fluent or contains errors. The research will include five thrusts: (1) based on ASL linguistics and pedagogy, to identify a set of observable attributes indicating ASL fluency; (2) to discover new technologies for automatic detection of the ASL fluency attributes through fusion of multimodality (facial expression, hand gesture, and body pose) and multisensory information (RGB and Depth videos); (3) to collect and annotate a dataset of RGBD videos of ASL, performed at varied levels of fluency, by students and native signers; (4) to develop an interactive ASL learning tool that provides ASL students immediate feedback about whether their signing is fluent or not; and (5) to evaluate the robustness of the new algorithms and the effectiveness of the ASL learning tool, including its educational benefits. The work will lead to advances in computer vision technologies for human behavior perception, to new understanding of user-interface design with ASL video, and to a revolutionary and cost-effective educational tool to assist ASL learners achieve fluency, using recognition technologies that are robust and accurate in the near-term. Project outcomes will include a dataset of videos at varied fluency levels, which will be valuable for future ASL linguists or instructors, students learning ASL, and computer vision researchers.

美国手语(ASL)是美国50万人的主要交流方式，也是一种有别于英语的语言，通过手、面部表情和身体动作来传达。研究表明，聋人父母的聋儿比聋人父母的聋儿阅读更好，这主要是因为当孩子和父母都是聋人时，沟通更好。然而，80%以上的聋哑或重听儿童是听力正常的父母所生。对于失聪儿童的父母、老师和生活中的其他人来说，足够快地学习ASL以支持孩子的视觉语言习得是具有挑战性的。能够自动识别美国手语签名的各个方面并向这些美国手语学生提供即时反馈的技术，将为他们提供一种灵活的时间方式来练习和提高他们的签名技能。该项目涉及纽约市立大学(CUNY)内三所学院的研究人员组成的跨学科团队，他们拥有计算机视觉、人机交互以及聋人和重听教育方面的专业知识，其目标是发现最有效的基础技术、用户界面设计和交互工具的教学使用，为ASL学生提供这种即时、自动的反馈。大多数先前关于ASL识别的工作集中于识别一小部分执行的简单手势，但目前的技术在用不受限制的词汇连续签署句子方面还不够准确。PIS将开发从根本上推进ASL部分识别的技术，即识别ASL的语言/性能属性，而不必识别整个手势序列，并自动确定表演是否流畅或包含错误。这项研究将包括五个方面：(1)基于ASL语言学和教育学，确定一组反映ASL流利性的可观察属性；(2)通过融合多模式(面部表情、手势和身体姿势)和多感官信息(RGB和深度视频)，发现自动检测ASL流利性属性的新技术；(3)收集和注释由学生和本族语签名者在不同流利度下进行的ASL RGBD视频集；(4)开发一个交互式ASL学习工具，为ASL学生提供关于其手语是否流利的即时反馈；以及(5)评估新算法的稳健性和ASL学习工具的有效性，包括其教育效益。这项工作将导致用于人类行为感知的计算机视觉技术的进步，通过ASL视频对用户界面设计的新理解，以及一种革命性的、具有成本效益的教育工具，以帮助ASL学习者实现流利，使用在短期内强大和准确的识别技术。项目成果将包括不同流利程度的视频数据集，这对未来的ASL语言学家或教师、学习ASL的学生和计算机视觉研究人员将是有价值的。