RI: Medium: Collaborative Research: Models of Handshape Articulatory Phonology for Recognition and Analysis of American Sign Language

手语是世界上数百万聋人的主要交流方式，其中包括美国约35万至50万美国手语（ASL）用户。虽然听力正常的人群已经从语音识别和语音网络搜索等语音技术的进步中受益，但在手语界面方面取得的进展要少得多。进步取决于从视频中分析手语的技术的改进。此外，人们对手语语言学的了解还不如对口语语言学的了解多。该项目通过跨学科的方法解决了这两个需求，将有助于语言学、语言处理、计算机视觉和机器学习的研究。这项工作的应用包括更好地访问美国手语社交媒体视频档案，聋人互动识别和搜索应用，以及美国手语英语口译协助。这个项目的重点是手语中的手形，特别是在一个受限但非常实用的组成部分：手指拼写，或拼写出一个单词的一系列手形和它们之间的轨迹。根据上下文的不同，手指拼写占美国手语的35%，包括72%的美国手语手型，这使它成为一个很好的试验场。该项目通过关注各种条件下的手部形状，包括快速、高度连贯的签名，解决了现有工作中的空白。主要的项目活动包括：(1)利用新的手形模型，包括ASL语音特征的分词和“多分词”图形模型，对手指拼写单词进行鲁棒自动检测和识别；(2)跨签名者、风格和记录条件的泛化技术；(3)改进了手形的语音和音韵学，特别是有助于形成符号的发音音韵学；(4)公开发布多说话人、多风格的指纹拼写数据及其半自动标注。

Sign languages are the primary means of communication for millions of Deaf people in the world, including about 350,000-500,000 American Sign Language (ASL) users in the US. While the hearing population has benefited from advances in speech technologies such as speech recognition and spoken web search, much less progress has been made for sign language interfaces. Advances depend on improved technology for analyzing sign language from video. In addition, the linguistics of sign language is less well-understood than that of spoken language. This project addresses both of these needs, with an interdisciplinary approach that will contribute to research in linguistics, language processing, computer vision, and machine learning. Applications of the work include better access to ASL social media video archives, interactive recognition and search applications for Deaf individuals, and ASL-English interpretation assistance.This project focuses on handshape in ASL, in particular on one constrained but very practical component: fingerspelling, or the spelling out of a word as a sequence of handshapes and trajectories between them. Fingerspelling comprises up to 35% of ASL, depending on the context, and includes 72% of ASL handshapes, making it an excellent testing ground. The project addresses gaps in existing work by focusing on handshape in various conditions, including fast, highly coarticulated signing. The main project activities include development of (1) robust automatic detection and recognition of fingerspelled words using new handshape models, including segmental and "multi-segmental" graphical models of ASL phonological features; (2) techniques for generalizing across signers, styles, and recording conditions; (3) improved phonetics and phonology of handshape, in particular contributing to an articulatory phonology of sign; and (4) publicly released multi-speaker, multi-style fingerspelling data and associated semi-automatic annotation.