RI: Medium: Collaborative Research: Models of Handshape Articulatory Phonology for Recognition and Analysis of American Sign Language

手语是世界上数百万聋人的主要交流手段，其中包括美国约35万至50万美国手语（ASL）用户。 虽然听力人群受益于语音识别和语音网络搜索等语音技术的进步，但手语界面的进步要少得多。 进步取决于从视频中分析手语的改进技术。 此外，手语的语言学不如口语那么容易理解。 该项目解决了这两个需求，采用跨学科的方法，将有助于语言学，语言处理，计算机视觉和机器学习的研究。 这项工作的应用包括更好地访问美国手语社交媒体视频档案，聋人个人的交互式识别和搜索应用程序，以及美国手语-英语口译援助。这个项目的重点是美国手语中的手形，特别是一个受限制但非常实用的组成部分：手指拼写，或拼写出一个单词作为一系列手形和它们之间的轨迹。 根据上下文，手指拼写包括高达35%的ASL，并包括72%的ASL手形，使其成为一个很好的测试场。 该项目通过关注各种条件下的手部形状（包括快速、高度共同表达的手语）来解决现有工作中的差距。 主要项目活动包括：（1）利用新的手形模型，包括美国手语语音特征的分段和“多分段”图形模型，开发强有力的手指拼写单词自动检测和识别技术;（2）跨手语者、风格和录音条件进行归纳的技术;（3）改进手形的语音和音系学，特别是有助于手语的发音音系学;以及（4）公开发布的多说话者、多风格的手指拼写数据和相关联的半自动注释。

Sign languages are the primary means of communication for millions of Deaf people in the world, including about 350,000-500,000 American Sign Language (ASL) users in the US. While the hearing population has benefited from advances in speech technologies such as speech recognition and spoken web search, much less progress has been made for sign language interfaces. Advances depend on improved technology for analyzing sign language from video. In addition, the linguistics of sign language is less well-understood than that of spoken language. This project addresses both of these needs, with an interdisciplinary approach that will contribute to research in linguistics, language processing, computer vision, and machine learning. Applications of the work include better access to ASL social media video archives, interactive recognition and search applications for Deaf individuals, and ASL-English interpretation assistance.This project focuses on handshape in ASL, in particular on one constrained but very practical component: fingerspelling, or the spelling out of a word as a sequence of handshapes and trajectories between them. Fingerspelling comprises up to 35% of ASL, depending on the context, and includes 72% of ASL handshapes, making it an excellent testing ground. The project addresses gaps in existing work by focusing on handshape in various conditions, including fast, highly coarticulated signing. The main project activities include development of (1) robust automatic detection and recognition of fingerspelled words using new handshape models, including segmental and "multi-segmental" graphical models of ASL phonological features; (2) techniques for generalizing across signers, styles, and recording conditions; (3) improved phonetics and phonology of handshape, in particular contributing to an articulatory phonology of sign; and (4) publicly released multi-speaker, multi-style fingerspelling data and associated semi-automatic annotation.