CAREER: Large Vocabulary Gesture Recognition for Everyone: Gesture Modeling and Recognition Tools for System Builders and Users

PI在这个项目中的目标是开发新的方法来自动注释，识别和索引手势的大词汇表，并使用这些方法来创建一套集成的手语识别工具。 用于识别大的手势词汇表的当前最先进的方法具有影响系统设计和用户体验两者的显著限制。 许多方法假设存在近乎完美的手部检测器/跟踪器;这是一个限制性的假设，这阻止了这些方法在复杂的现实世界环境中的部署，在复杂的现实世界中，这种准确性是无法实现的。 在没有完美的手检测器的情况下，系统设计可能涉及对训练视频的手动注释的大量投资（例如，指定手的位置），以便向训练模块提供足够干净的信息。 用户体验受到现有应用程序有限的准确性和鲁棒性的影响。 在这项研究中，PI将通过明确设计识别和索引方法来解决这些问题，这些方法既不需要完美的手部检测器，也不需要大量的手动注释，从而使在现实世界中部署准确有效的手势识别系统变得更加容易。 PI将通过计算机视觉，模式识别和数据库索引的当前最先进技术的理论进步来实现这些目标。 该项目的统一主题是将产生不完美输出的低级跟踪模块与识别和索引方法集成，这些方法旨在将跟踪模块的不完美输出作为输入。 将开发新的铰接式跟踪方法，利用概率图模型提供全自动的长期跟踪，同时改善概率图模型目前产生的过度时间复杂性。 新的方法将被设计用于从手部外观中提取和利用信息。 由于这些新的建模和识别方法将违反现有索引方法所做的标准假设，因此将制定新的索引方法，这将提高在拟议框架内的动态手势和静态手部形状的大型数据库中的搜索效率。项目成果将显著提高世界各地手语用户搜索手语视频数据库的能力，例如查找未知标志的含义或检索连续签名视频中感兴趣的标志的出现。 这些搜索工具将在教育环境中产生影响，促进学习手语和获取手语提供的任意信息。 为此，PI将在网上向公众免费提供他的软件。 他还将与美国手语专家合作，使用他的工具实现关键应用程序，这些工具将提供给聋人学生。 PI还将开发一个公开的手势识别源代码、应用程序和数据集包，以帮助各级学生研究人员从事手势识别研究。 作为一项旨在吸引年轻人从事科学职业的额外外联活动，PI将共同组织夏令营，对初中和高中学生进行计算机科学教育。

The PI's goal in this project is to develop new methods for automatically annotating, recognizing, and indexing large vocabularies of gestures, and to use these methods to create an integrated set of tools for sign language recognition. Current state-of-the-art methods for recognizing large vocabularies of gestures have significant limitations that impact both system design and the user experience. Many methods assume the existence of a near-perfect hand detector/tracker; that is a limiting assumption, which prevents deployment of these methods in complex real-world settings where such accuracy is unachievable. In the absence of perfect hand detectors, system design may involve a large investment in manual annotation of training videos (e.g., specifying hand locations), so as to provide sufficiently clean information to training modules. The user experience is affected by the limited accuracy and robustness of existing applications. In this research the PI will address these issues by explicitly designing recognition and indexing methods that require neither perfect hand detectors nor extensive manual annotations, thus making it substantially easier to deploy accurate and efficient gesture recognition systems in real-world settings. The PI will achieve these objectives through theoretical advances in the current state of the art in computer vision, pattern recognition, and database indexing. The unifying theme in the project is the integration of low-level tracking modules that produce imperfect output, with recognition and indexing methods that are designed to take as input this imperfect output from the tracking modules. Novel articulated tracking methods will be developed that utilize probabilistic graph models to provide fully automatic long-term tracking, while improving upon the excessive time complexity that probabilistic graph models currently incur. New methods will be designed for extracting and exploiting information from hand appearance. As these novel modeling and recognition methods will violate standard assumptions made by existing indexing methods, new indexing methods will be formulated which will improve the efficiency of search in large databases of dynamic gestures and static hand shapes within the proposed framework.Broader Impacts: Project outcomes will significantly improve the ability of sign language users around the world to search databases of sign language videos and to perform tasks such as looking up the meaning of an unknown sign or retrieving occurrences of a sign of interest in videos of continuous signing. These search tools will have an impact in educational settings, facilitating both learning a sign language and accessing arbitrary information available in a sign language. To these ends, the PI will make his software freely available to the public online. He will also work with experts in American Sign Language to implement key applications using his tools, which will be made available to Deaf students. The PI will furthermore develop a publicly available package of gesture recognition source code, applications, and datasets that will help student researchers at all levels engage in gesture recognition research. As an additional outreach activity intended to attract young people to careers in science, the PI will co-organize summer camps that educate junior high and high school students in computer science.