Collaborative Research: HCC: Medium: Linguistically-Driven Sign Recognition from Continuous Signing for American Sign Language (ASL)

目前还没有方法从连续签名的视频中分割和识别符号。在这个项目中，为美国手语（ASL）开发的基于计算机的符号识别技术将从孤立的、引用形式的符号扩展到句子中的符号。连续手势的识别是一项具有挑战性的任务，部分原因是一个手势的发音经常受到相邻手势的影响。最近的许多深度学习研究都集中在使用“未分割”方法的连续手势短视频上，在这种方法中，视频中的单词在不检测每个手势的开始和结束时间的情况下被识别。然而，美国手语话语通常包含不同类型的符号（如词汇、手指拼写、分类器结构），这些符号具有显著不同的内部结构，需要不同的识别策略。因此，识别包含不同符号类型的视频子持续时间的“分段”识别对于完整的识别体系结构是必要的。分段识别还可以自动对ASL视频进行时间戳注释，为人工智能研究提供训练数据，同时还可以为ASL学习者（通常在分析连续签名时遇到困难）提供强大的工具，对视频进行分词分析。新方法和本项目收集的所有数据将公开共享，促进计算机视觉、图形学、人机交互、美国手语、语言学和其他相关科学的新研究。这项研究还将为各种各样的技术铺平道路，以改善聋人和听力健全者之间的交流，例如：美国手语到英语的翻译（手语识别是其先驱）；支持美国手语学习者的教育应用；以及在网络视频上进行类似谷歌的符号搜索。项目成果将产生更广泛的影响，因为这些相同的技术可以应用于其他手语，因为新方法将被纳入pi机构的教育计划。除了这些对研究和教育的社会影响和利益之外，ppi将继续他们长期以来的传统，招募失聪或听力障碍的学生，以及其他未被充分代表的群体成员，参与这项研究。该项目将开发一种新颖的端到端机器学习方法，其中包括以下关键组件：(1)从包含不同类型ASL手势的连续签名中分割区域（基于从特定帧数的窗口提取的2D骨架数据，使用AlphaPose）；(2)使用时空GCN方法分割区域内的单个标志；(3)对分割后的符号进行识别（使用变压器进行符号分类）。分割符号的识别将利用适用于识别的符号类型的语言约束，本项目将重点放在词汇符号上，并将考虑协同发音效应。为此，研究将建立在pi的大型，公开共享的，语言注释的，孤立符号和连续符号的视频语料库上。这种方法具有明确的符号类型检测、符号分割和针对分段符号的特定类型识别策略，对于大规模成功的连续符号识别和广泛的应用至关重要。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

There is currently no method to segment and recognize signs from videos of continuous signing. In this project, computer-based sign recognition techniques developed for American Sign Language (ASL) will be extended from isolated, citation-form signs to signs in sentences. Sign recognition from continuous signing is a challenging task, in part because the articulation of one sign is often affected by that of neighboring signs. Much of the recent deep-learning research has focused on short videos of continuous signing using an “unsegmented” approach, in which the words in a video are identified without detection of start and end times for each sign. However, ASL utterances typically contain signs of different types (e.g., lexical, fingerspelled, classifier constructions) that have significantly different internal structures, requiring distinct recognition strategies. Thus, “segmented” recognition to identify video sub-durations containing distinct sign types is necessary for a complete recognition architecture. Segmented recognition would also enable automatic time-stamped annotation of ASL videos to produce training data for AI research, as well as powerful tools to provide ASL learners (who often have trouble parsing continuous signing) with a word-segmented analysis of videos. The new methods and all data collected for this project will be shared publicly, facilitating new research in computer vision, graphics, HCI, ASL, linguistics, and other related sciences. The research will also pave the way for a wide variety of technologies to improve communication between deaf and hearing individuals, such as: ASL-to-English translation (for which sign recognition is a precursor); educational applications to support ASL learners; and Google-like sign search by example over videos on the Web. Additional broad impact will derive from project outcomes because these same technologies can be applied to other signed languages, and because the new methods will be incorporated into educational programs in the PIs’ institutions. Beyond these societal impacts and benefits for research and education, the PIs will continue their long tradition of recruiting students who are deaf or hard-of-hearing, as well as members of other underrepresented groups, to participate in this research.This project will develop a novel end-to-end machine learning approach with the following key components: (1) segmentation of regions from continuous signing that contain distinct types of ASL signs (based on 2D skeleton data extracted from a window of a specific number of frames, using AlphaPose); (2) segmentation of the individual signs within those regions (using a spatiotemporal GCN approach); and (3) recognition of the segmented signs (using a transformer for sign classification). The recognition of segmented signs will leverage linguistic constraints applicable to the recognized sign type, with a focus in this project on lexical signs, and will also consider coarticulation effects. To these ends, the research will build on the PIs’ large, publicly shared, linguistically annotated, video corpora of isolated signs and continuous signing. This approach, with explicit sign type detection, sign segmentation, and type-specific recognition strategies for segmented signs, is essential for successful continuous sign recognition at scale and for a wide range of applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.