III: Medium: Collaborative Research: Linguistically Based ASL Sign Recognition as a Structured Multivariate Learning Problem

语言在空间中的表现对计算机识别提出了特殊的挑战。先前的符号识别方法没有利用语言结构和约束的知识，部分原因是所采用的计算模型的局限性。此外，他们还专注于识别有限类别的符号。没有一个系统能够识别所有的语音类型的符号，或者甚至可以在连续的手语中区分这些符号。通过整合几种计算方法，了解手语的语言特性，并在现有的大型语言注释语料库的支持下，该团队将开发一个强大的、全面的框架，用于从自然、连续的手语视频流中识别手语。符号语言结构的根本差异，将四维手语与口语区分开来，具有时空依赖性和多种生产渠道，对基于计算机的识别至关重要。这是因为手指拼写的项目，词汇符号和分类器结构，例如，需要不同的识别策略。语言特性将在这里被用于(i)对明显不同类型的符号进行分割和分类，然后，尽管这个后续的企业在项目期间的范围必然受到限制，（ii）对分割的符号序列的识别。通过团队开发的跟踪器进行三维手部姿态估计，可以获得显著的跟踪精度、鲁棒性和计算效率。与仅使用2D信息的识别方案相比，这种3D信息有望大大改善识别结果。来自跟踪的3D估计信息将用于提出的基于分层条件随机场（CRF）的识别，以允许跟踪和识别在其语言组成上不同的符号。由于其他手语也依赖于非常相似的符号类型，因此该技术将很容易扩展到基于计算机的其他手语识别中。这种基于语言的分层框架的手语识别？基于直接适用于其他手语的技术？首次提供了一种建模和分析手语的离散和连续方面的方法，也使适当的识别策略能够应用于语言不同组成的手语。这种方法还将允许未来将面部手势的离散和连续方面与手动签名相结合，以进一步改进基于计算机的美国手语建模和分析。缺乏这样一个框架阻碍了手语的识别和生成。这一领域的进步将反过来对普及和改善与聋人的沟通产生深远的好处。该技术的进一步应用包括计算机对一般非语言交流的自动识别和分析、安全应用、人机界面以及虚拟和增强现实。事实上，这些技术对于任何具有连续和离散方面的以人为中心的应用程序都有潜在的实用价值。所提出的方法将为解决其他领域的类似问题提供方法，这些领域以多维和复杂的时空数据为特征，需要结合领域知识。这项研究的产品，包括软件、视频和注释，将在研究和教育中公开使用。

The manifestation of language in space poses special challenges for computer-based recognition. Prior approaches to sign recognition have not leveraged knowledge of linguistic structures and constraints, in part because of limitations in the computational models employed. In addition, they have focused on the recognition of limited classes of signs. No system exists that can recognize signs of all morphophonological types or that can even discriminate among these in continuous signing. Through integration of several computational approaches, informed by knowledge of linguistic properties of manual signs, and supported by a large existing linguistically annotated corpus, the team will develop a robust, comprehensive framework for sign recognition from video streams of natural, continuous signing. Fundamental differences in the linguistic structure of signs, distinguishing signed languages in 4D, with spatio-temporal dependencies and multiple production channels from spoken languages, are critical to computer-based recognition. This is because finger-spelled items, lexical signs, and classifier constructions, e.g., require different recognition strategies. Linguistic properties will be leveraged here for (i) segmentation and categorization of significantly different types of signs, and then, although this subsequent enterprise will necessarily be limited in scope within the project period, (ii) recognition of the segmented sign sequences. Through the 3D hand pose estimation from a team-developed tracker, w significant tracking accuracy, robustness, and computational efficiency will be attained. This 3D information is expected to greatly improve the recognition results, as compared with recognition schemes using only 2D information. The 3D estimated information from the tracking will be used in the proposed hierarchical Conditional Random Field (CRF) based recognition, to allow for tracking and recognition of signs that are distinct in their linguistic composition. Since other signed languages also rely on a very similar sign typology, this technology will be readily extensible to computer-based recognition of other signed languages.This linguistically-based hierarchical framework for ASL sign recognition?based on techniques with direct applicability to other signed languages, as well?provides, for the first time, a way to model and analyze the discrete and continuous aspects of signing, also enabling appropriate recognition strategies to be applied to signs with linguistically different composition. This approach will also allow the future integration of the discrete and continuous aspects of facial gestures with manual signing, to further improve computer-based modeling and analysis of ASL. The lack of such a framework has held back sign language recognition and generation. Advances in this area will, in turn, have far-ranging benefits for Universal Access and improved communication with the Deaf. Further applications of this technology include automated recognition and analysis by computer of non-verbal communication in general, security applications, human-computer interfaces, and virtual and augmented reality. In fact, these techniques have potential utility for any human-centered applications with continuous and discrete aspects. The proposed approach will offer ways to address similar problems in other domains characterized by multidimensional and complex spatio-temporal data that require the incorporation of domain knowledge. The products of this research, including software, videos, and annotations, will be made publicly available for use in research and education.