HCC: Large Lexicon Gesture Representation, Recognition, and Retrieval

据估计，美国手语(ASL)在美国有多达200万人使用。然而，由于ASL的可视性质和缺乏标准的书面形式，许多被口语用户视为理所当然的资源对ASL用户来说是不可用的。例如，当ASL用户遇到未知符号时，不能在词典中查找它。在现有的ASL词典中，人们可以很容易地找到与英语单词相对应的手势，但不能找到与给定手势相对应的英语单词(或者更广泛地说，是什么意思)。另一个例子是使用关键字搜索计算机文件或网页，这现在是计算机用户的一项频繁活动。目前，还没有ASL的关键字搜索的等价物。ASL不是一种书面语言，与文本文档最接近的等价物是ASL叙述或交流的视频序列。目前还没有工具可用于查找出现特定标志的视频片段。此类工具的缺乏严重限制了对ASL文献、知识、诗歌、表演或课程的视频库的基于内容的访问。这项研究的核心目标是通过推进基于视觉的手势识别和检索的最新技术，推动这些资源的可用。这在计算机视觉、机器学习和数据库索引等领域提出了具有挑战性的研究问题。这方面的工作将集中在以下方面：通过将手势分解为语音元素，开发每个手语只有几个训练样本的手语类学习模型的方法；为手势语言的视频词典设计可伸缩的索引方法，以在存在数千个手势类的情况下以交互速度实现手势识别；创建用于识别美国手语视频数据库中出现的手势的索引方法；纳入语言约束以提高较低级别视觉模块的性能，例如手势估计和上半身跟踪以及高级学习和索引模块；以及明确设计能够与往往提供不准确或模棱两可的输出的容易出错的视觉模块一起工作的方法。PIS将创建两个演示系统：一个是美国手语词典，其中包含美国手语手语的全面数据库；另一个是“手语谷歌”，它可以在美国手语视频内容的大型数据库中搜索特定手语。这些系统将使用数以千计的视频序列进行培训和评估，这些视频序列是由当地的ASL签名者在隔离和上下文中执行的。这些使用数据将对研究协同发音效应和上下文相关的手势变化很有价值。收集的手势将包括标准大学ASL课程头三年出现的ASL手势的完整清单。广泛影响：该项目开发的方法将使基于手势的搜索ASL文献、知识、诗歌、表演、课程、数字视频图书馆和DVD，这一能力将对改善聋人的教育、机会和机会产生深远影响。这些算法还旨在实现对ASL词典的基于视频的查询，并最终实现具有关于符号和用法实例的元语言信息的成熟词典。通过使那些学习ASL的人能够“查看”他们不知道的手势，这项技术有望改变ASL的学生(聋人和听力)、聋人儿童的父母、手语翻译和语言学家学习他们遇到的手势的方式。在这项工作中开发的算法很可能导致更强大的ASL识别系统，它可以处理具有大量手势词典的自然手势，该技术还将推动手势识别和合成系统的最先进水平。作为这一努力的一部分，产生的大量带有语言注释的母语ASL语料库本身将是一项重要的资源。

It is estimated that American Sign Language (ASL) is used by up to 2 million people in the United States. Yet many resources that are taken for granted by users of spoken languages are not available to users of ASL, given its visual nature and its lack of a standard written form. For instance, when an ASL user encounters an unknown sign, looking it up in a dictionary is not an option. With existing ASL dictionaries one can easily find what sign corresponds to an English word, but not what English word (or, more generally, what meaning) corresponds to a given sign. Another example is searching for computer files or web pages using keywords, which is now a frequent activity for computer users. At present, no equivalent for keyword search exists for ASL. ASL is not a written language, and the closest equivalent of a text document is a video sequence of ASL narration or communication. No tools are currently available for finding video segments in which specific signs occur. The lack of such tools severely restricts content-based access to video libraries of ASL literature, lore, poems, performances, or courses. The core goal of this research is to push towards making such resources available, by advancing the state-of-the-art in vision-based gesture recognition and retrieval. This poses challenging research problems in the areas of computer vision, machine learning, and database indexing. The effort will focus on the following: developing methods for learning models of sign classes, given only a few training examples per sign, by using a decomposition of signs into phonological elements; designing scalable indexing methods for video lexicons of gestural languages that achieve sign recognition at interactive speeds, in the presence of thousands of classes; creating indexing methods for spotting signs appearing in context in an ASL video database; incorporating linguistic constraints to improve performance of both lower-level vision modules, such as hand pose estimation and upper body tracking, and higher-level learning and indexing modules; and explicitly designing methods that can work with error-prone vision modules that often provide inaccurate or ambiguous outputs. The PIs will create two demonstration systems: an ASL lexicon containing a comprehensive database of ASL signs; and a "Sign Language Google" that can search for specific signs in large databases of ASL video content. The systems will be trained and evaluated using thousands of video sequences of signs performed in isolation and in context by native ASL signers. This usage data will be valuable for studying co-articulation effects and context-dependent sign variations. The signs collected will include the full list of ASL signs appearing in the first three years of standard college ASL curricula.Broader Impacts: The methods developed in this project will enable sign-based search of ASL literature, lore, poems, performances, courses, from digital video libraries and DVDs, a capability which will have far-reaching implications for improving education, opportunities, and access for the deaf. These algorithms also aim to enable video-based queries of ASL lexicons, and eventually full-fledged dictionaries with metalinguistic information about signs and examples of usage. By enabling those learning ASL to "look up" a sign they do not know, this technology promises to transform the way students of ASL (both deaf and hearing), parents of deaf children, sign language interpreters, and linguists learn about signs they encounter. The algorithms developed in this effort may well lead to more robust ASL recognition systems, which can handle natural signing with a large lexicon of signs and the technology will also advance the state of the art in gesture recognition and synthesis systems. The large linguistically annotated corpus of native ASL produced as part of this effort will itself be an important resource.