III: Medium: Collaborative Research: Linguistically Based ASL Sign Recognition as a Structured Multivariate Learning Problem

语言在空间的表现形式对基于计算机的识别提出了特殊的挑战。以前的手势识别方法没有利用语言结构和约束的知识，部分原因是所采用的计算模型的局限性。此外，他们还将重点放在识别有限类别的标志上。没有一种系统可以识别所有形态音素类型的符号，甚至可以在连续手势中区分这些符号。通过整合几种计算方法，了解手工手势的语言特性，并在现有的大型语言注释语料库的支持下，该小组将开发一个强大的、全面的框架，用于从自然、连续手势的视频流中识别手势。手语在语言结构上的根本差异，在4D中区分手语，具有时空依赖性和与口语的多个产生渠道，是基于计算机的识别的关键。这是因为手指拼写的项目、词汇符号和分类器结构例如需要不同的识别策略。这里将利用语言特性来(I)对显著不同类型的标志进行分割和分类，然后(Ii)识别分割后的标志序列，尽管这一后续项目的范围必然在项目期间内受到限制。通过团队开发的跟踪器进行3D手势估计，将获得显著的跟踪精度、鲁棒性和计算效率。与仅使用2D信息的识别方案相比，这种3D信息有望极大地提高识别结果。来自跟踪的3D估计信息将被用于所提出的基于分层条件随机场(CRF)的识别，以允许跟踪和识别在其语言组成上不同的符号。由于其他手语也依赖于非常相似的手语类型，这项技术将很容易扩展到基于计算机的其他手语识别。这种基于语言的ASL手语识别层次框架-基于直接适用于其他手语的技术-首次提供了一种建模和分析手语离散和连续方面的方法，也使得适当的识别策略能够应用于具有不同语言组成的手语。这种方法还将允许未来将面部手势的离散和连续方面与手动手势相结合，以进一步改进基于计算机的ASL建模和分析。缺乏这样的框架阻碍了手语的识别和生成。反过来，这一领域的进步将对普遍接入和改善与聋人的交流产生广泛的好处。这项技术的进一步应用包括计算机对一般非语言交流的自动识别和分析、安全应用、人机界面以及虚拟和增强现实。事实上，这些技术对于任何以人为中心、具有连续和离散方面的应用程序都具有潜在的实用价值。所提议的方法将为解决其他领域的类似问题提供方法，这些领域以多维和复杂的时空数据为特征，需要纳入领域知识。这项研究的产品，包括软件、视频和注释，将公开用于研究和教育。

The manifestation of language in space poses special challenges for computer-based recognition. Prior approaches to sign recognition have not leveraged knowledge of linguistic structures and constraints, in part because of limitations in the computational models employed. In addition, they have focused on the recognition of limited classes of signs. No system exists that can recognize signs of all morphophonological types or that can even discriminate among these in continuous signing. Through integration of several computational approaches, informed by knowledge of linguistic properties of manual signs, and supported by a large existing linguistically annotated corpus, the team will develop a robust, comprehensive framework for sign recognition from video streams of natural, continuous signing. Fundamental differences in the linguistic structure of signs, distinguishing signed languages in 4D, with spatio-temporal dependencies and multiple production channels from spoken languages, are critical to computer-based recognition. This is because finger-spelled items, lexical signs, and classifier constructions, e.g., require different recognition strategies. Linguistic properties will be leveraged here for (i) segmentation and categorization of significantly different types of signs, and then, although this subsequent enterprise will necessarily be limited in scope within the project period, (ii) recognition of the segmented sign sequences. Through the 3D hand pose estimation from a team-developed tracker, w significant tracking accuracy, robustness, and computational efficiency will be attained. This 3D information is expected to greatly improve the recognition results, as compared with recognition schemes using only 2D information. The 3D estimated information from the tracking will be used in the proposed hierarchical Conditional Random Field (CRF) based recognition, to allow for tracking and recognition of signs that are distinct in their linguistic composition. Since other signed languages also rely on a very similar sign typology, this technology will be readily extensible to computer-based recognition of other signed languages.This linguistically-based hierarchical framework for ASL sign recognition?based on techniques with direct applicability to other signed languages, as well?provides, for the first time, a way to model and analyze the discrete and continuous aspects of signing, also enabling appropriate recognition strategies to be applied to signs with linguistically different composition. This approach will also allow the future integration of the discrete and continuous aspects of facial gestures with manual signing, to further improve computer-based modeling and analysis of ASL. The lack of such a framework has held back sign language recognition and generation. Advances in this area will, in turn, have far-ranging benefits for Universal Access and improved communication with the Deaf. Further applications of this technology include automated recognition and analysis by computer of non-verbal communication in general, security applications, human-computer interfaces, and virtual and augmented reality. In fact, these techniques have potential utility for any human-centered applications with continuous and discrete aspects. The proposed approach will offer ways to address similar problems in other domains characterized by multidimensional and complex spatio-temporal data that require the incorporation of domain knowledge. The products of this research, including software, videos, and annotations, will be made publicly available for use in research and education.