RI: Medium: Collaborative Research: Models of Handshape Articulatory Phonology for Recognition and Analysis of American Sign Language

手语是世界上数百万聋人的主要交流手段，其中包括美国约35万至50万美国手语用户。虽然听力人口受益于语音识别和语音网络搜索等语音技术的进步，但手语界面的进步要小得多。进步取决于从视频中分析手语的改进技术。此外，手语的语言学没有口语那么容易理解。这个项目解决了这两个需求，采用了跨学科的方法，将有助于语言学、语言处理、计算机视觉和机器学习的研究。这项工作的应用包括更好地访问ASL社交媒体视频档案，为聋人提供交互式识别和搜索应用程序，以及ASL-英语口译辅助。本项目关注ASL中的手形，特别是一个受限制但非常实用的组成部分：指纹拼写，即将单词拼写出它们之间的一系列手形和轨迹。根据上下文的不同，指纹拼写在美国手语中的比例高达35%，在美国手形中的比例为72%，这使它成为一个很好的试验场。该项目通过专注于各种条件下的手形，包括快速、高度协同表达的签名，解决了现有工作中的空白。主要项目活动包括：(1)利用新的手形模型，包括美国手语语音特征的分段和“多段”图形模型，对手写单词进行稳健的自动检测和识别；(2)跨签名者、风格和记录条件推广的技术；(3)改进手形的语音和音系学，特别是有助于手势发音的音系学；(4)公开发布多说话人、多种风格的手指拼写数据和相关的半自动注释。

Sign languages are the primary means of communication for millions of Deaf people in the world, including about 350,000-500,000 American Sign Language (ASL) users in the US. While the hearing population has benefited from advances in speech technologies such as speech recognition and spoken web search, much less progress has been made for sign language interfaces. Advances depend on improved technology for analyzing sign language from video. In addition, the linguistics of sign language is less well-understood than that of spoken language. This project addresses both of these needs, with an interdisciplinary approach that will contribute to research in linguistics, language processing, computer vision, and machine learning. Applications of the work include better access to ASL social media video archives, interactive recognition and search applications for Deaf individuals, and ASL-English interpretation assistance.This project focuses on handshape in ASL, in particular on one constrained but very practical component: fingerspelling, or the spelling out of a word as a sequence of handshapes and trajectories between them. Fingerspelling comprises up to 35% of ASL, depending on the context, and includes 72% of ASL handshapes, making it an excellent testing ground. The project addresses gaps in existing work by focusing on handshape in various conditions, including fast, highly coarticulated signing. The main project activities include development of (1) robust automatic detection and recognition of fingerspelled words using new handshape models, including segmental and "multi-segmental" graphical models of ASL phonological features; (2) techniques for generalizing across signers, styles, and recording conditions; (3) improved phonetics and phonology of handshape, in particular contributing to an articulatory phonology of sign; and (4) publicly released multi-speaker, multi-style fingerspelling data and associated semi-automatic annotation.