Collaborative Research: HCC: Medium: Linguistically-Driven Sign Recognition from Continuous Signing for American Sign Language (ASL)

目前还没有从连续签名的视频中分割和识别签名的方法。 在这个项目中，为美国手语（ASL）开发的基于计算机的符号识别技术将从孤立的引用形式的符号扩展到句子中的符号。从连续的手势中识别手势是一项具有挑战性的任务，部分原因是一个手势的清晰度经常受到相邻手势的影响。最近的大部分深度学习研究都集中在使用“未分割”方法进行连续签名的短视频上，在这种方法中，视频中的单词被识别出来，而不需要检测每个符号的开始和结束时间。然而，ASL话语通常包含不同类型的符号（例如，词汇、指法、分类器结构）具有显著不同的内部结构，需要不同的识别策略。因此，“分段”识别，以识别视频子持续时间包含不同的标志类型是必要的一个完整的识别架构。分段识别还将使ASL视频的自动时间戳注释能够为人工智能研究产生训练数据，以及为ASL学习者（他们经常难以解析连续签名）提供视频的单词分段分析的强大工具。 新方法和为该项目收集的所有数据将公开共享，促进计算机视觉，图形学，HCI，ASL，语言学和其他相关科学的新研究。该研究还将为各种技术铺平道路，以改善聋人和听力个体之间的沟通，例如：ASL到英语的翻译（其中符号识别是先驱）;支持ASL学习者的教育应用程序;以及通过网络上的视频进行类似Google的符号搜索。 项目成果还将产生更广泛的影响，因为这些技术可以应用于其他手语，而且新方法将被纳入公共机构的教育方案。 除了这些社会影响以及对研究和教育的好处之外，PI将继续招募聋人或听力困难的学生以及其他代表性不足的群体的成员参与这项研究。该项目将开发一种新的端到端机器学习方法，其关键组成部分如下：（1）从包含不同类型的ASL符号的连续符号分割区域（基于从特定帧数的窗口提取的2D骨架数据，使用AlphaPose）;（2）分割这些区域内的各个体征（使用时空GCN方法）;以及（3）识别分割的体征（使用用于体征分类的Transformer）。分割标志的识别将利用适用于已识别标志类型的语言约束，重点是词汇标志，并将考虑协同发音效应。为此，研究将建立在PI的大型，公开共享，语言注释，孤立的标志和连续签名的视频语料库。这种方法，与明确的标志类型检测，标志分割，和特定类型的识别策略分割的标志，是必不可少的成功连续标志识别的规模和广泛的应用。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

There is currently no method to segment and recognize signs from videos of continuous signing. In this project, computer-based sign recognition techniques developed for American Sign Language (ASL) will be extended from isolated, citation-form signs to signs in sentences. Sign recognition from continuous signing is a challenging task, in part because the articulation of one sign is often affected by that of neighboring signs. Much of the recent deep-learning research has focused on short videos of continuous signing using an “unsegmented” approach, in which the words in a video are identified without detection of start and end times for each sign. However, ASL utterances typically contain signs of different types (e.g., lexical, fingerspelled, classifier constructions) that have significantly different internal structures, requiring distinct recognition strategies. Thus, “segmented” recognition to identify video sub-durations containing distinct sign types is necessary for a complete recognition architecture. Segmented recognition would also enable automatic time-stamped annotation of ASL videos to produce training data for AI research, as well as powerful tools to provide ASL learners (who often have trouble parsing continuous signing) with a word-segmented analysis of videos. The new methods and all data collected for this project will be shared publicly, facilitating new research in computer vision, graphics, HCI, ASL, linguistics, and other related sciences. The research will also pave the way for a wide variety of technologies to improve communication between deaf and hearing individuals, such as: ASL-to-English translation (for which sign recognition is a precursor); educational applications to support ASL learners; and Google-like sign search by example over videos on the Web. Additional broad impact will derive from project outcomes because these same technologies can be applied to other signed languages, and because the new methods will be incorporated into educational programs in the PIs’ institutions. Beyond these societal impacts and benefits for research and education, the PIs will continue their long tradition of recruiting students who are deaf or hard-of-hearing, as well as members of other underrepresented groups, to participate in this research.This project will develop a novel end-to-end machine learning approach with the following key components: (1) segmentation of regions from continuous signing that contain distinct types of ASL signs (based on 2D skeleton data extracted from a window of a specific number of frames, using AlphaPose); (2) segmentation of the individual signs within those regions (using a spatiotemporal GCN approach); and (3) recognition of the segmented signs (using a transformer for sign classification). The recognition of segmented signs will leverage linguistic constraints applicable to the recognized sign type, with a focus in this project on lexical signs, and will also consider coarticulation effects. To these ends, the research will build on the PIs’ large, publicly shared, linguistically annotated, video corpora of isolated signs and continuous signing. This approach, with explicit sign type detection, sign segmentation, and type-specific recognition strategies for segmented signs, is essential for successful continuous sign recognition at scale and for a wide range of applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.