ITR: Fundamental Issues in Automated American Sign Language Recognition

手语是复杂的、抽象的语言系统，有自己的语法，它们的“发音”不仅涉及手，还涉及脸、肩膀和手臂。 在这个项目中，PI和他的团队将推动可扩展的自动化美国手语（ASL）识别形式主义的最新发展。 目前，有一些方法来识别孤立的标志，并在某种程度上，连续的标志在短句从一个单一的签名，主要使用特殊的设备，如数据手套或磁性标记或从视觉输入对平原背景和特殊服装。 PI旨在在五个领域取得实质性进展：（i）对不同背景和不同服装的识别，（ii）使用非手动方面，例如面部表情和头部运动，（iii）跨签名者的识别，（iv）鲁棒的、可扩展的形式主义的设计，以及（v）开发运用诸如视点、背景、时间和签名者的变量的大型ASL数据语料库，帮助衡量进展。 为此，PI将（i）为ASL开发强大的手动和非手动（面部）特征集，（ii）构建形式主义以从例句中学习符号模型，（iii）调查符号的基本形式是否（符号）可以学习，（iv）采用基于贝叶斯网络的索引方案来限制识别的组合，及（v）探讨将语法和句法信息纳入识别过程的技术。随着逐渐转向基于语音的I/O设备用于人机交互，除非在手语自动识别方面取得重大进展，否则依赖手语进行交流的人将很有可能无法获得最先进的技术。 这些进步还将通过促进残疾人在机场和杂货店等公共场合与普通民众的互动，提高残疾人的生活质量。 PI将确定至少一名聋人研究生或沟通障碍学生参与项目，以确保结果与预期用户社区相关且适当。 在这个项目中收集的大型ASL数据语料库将被积极分发;在项目结束日期之后，这项工作将继续得到支持。

Sign languages are complex, abstract linguistic systems, with their own grammars, and their "articulation" involve not just the hands, but the face, shoulder, and arms. In this project the PI and his team will push the state of the art in scalable automated American Sign Language (ASL) recognition formalisms. Presently, there are methods to recognize isolated signs and, to some extent, continuous signs in short sentences from a single signer, mostly using special equipment such as data gloves or magnetic markers or from visual input against plain background and special clothing. The PI seeks to achieve substantial advances in five areas:: (i) recognition against varied backgrounds and different clothing, (ii) use of non-manual aspects such as facial expression and head movement, (iii) recognition across signers, (iv) design of robust, scalable formalisms, and (v) development of large ASL data corpus that exercise variates such as viewpoint, background, time, and signer, to help benchmark progress. To these ends, the PI will (i) develop robust manual and non-manual (face) feature sets for ASL, (ii) construct formalisms to learn sign models from example sentences, (iii) investigate if elemental forms of signs (signemes) can be learned, (iv) employ Bayesian network based indexing schemes to limit the combinatorics of recognition, and (v) explore techniques to incorporate grammar and syntax information into the recognition process.Broader Impacts: With the gradual shift to speech based I/O devices for human computer interaction, there is great danger that people who rely on sign languages for communication will be deprived access to state of the art technology unless there significant advances in automated recognition of sign language are achieved. Such advances will also enhance the quality of life of persons with disabilities, by facilitating interaction with the general populace in public situations, such as airports and grocery stores. The PI will identify at least one deaf graduate student or a student with communication disorder to participate in the project, to ensure the outcome is relevant and appropriate to the intended user community. The large ASL data corpus collected in this project will be distributed aggressively; this effort will continue to be supported beyond the project end date.