Pattern Discovery in Signed Languages and Gestural Communication

识别和生成的手语和口语的手势组成部分的研究一直受到阻碍的大规模的语言注释语料库，导致在口语领域的重大进展。 此外，在手部、面部和上半身同时表达语言信息的复杂性为语言分析和基于计算机的识别带来了特殊的挑战。 该项目的一个主要目标是开发模式分析算法，用于发现手、脸和上身的语言学上重要的运动的共现、重叠、相对时间、频率和幅度。 这些将被测试对PI的最近开发的语料库收集从美国手语（ASL）的本地签名。高品质的视频数据包括多个同步的电影？les，从多个角度显示签名（包括面部特写）。 使用SignStream（PI开发的应用程序）生成注释，该应用程序能够转录并行信息流（例如，手、眼睛、眉毛等的动作，用手语传达重要的语法信息）。 视频数据和注释提供了用于分析在多个手动和非手动通道中发生的姿势的基础。 目标是识别通道内和通道间的时间关联。将开发时间序列分析算法用于比较手势组件的相似程度。将开发聚类和索引算法用于从混合离散注释事件标签（例如，光泽度、眉毛提升、手形等）以及采样的测量数据（例如，头部定向、眼睛注视方向、手部运动）。 由于若干手势通道包括变化频率和幅度的周期性运动（例如，摇头和摇头），还将开发周期性分析模块。 语言学家和计算机科学家将合作确定如何最好地利用和联合收割机从这些不同的来源提供的信息。 此外，从计算机科学研究中获得的信息将对PI研究小组正在进行的关于ASL语法的语言学研究有巨大的好处。 手语的句法研究一直受到一项艰巨的任务的阻碍，那就是试图仅仅通过观察来发现这些模式，而不借助分析大量数据的工具。为了支持SignStream中的时间序列模式分析，将开发用于分析视频的计算机视觉算法，以提取头部，面部，眼睛/眉毛，手臂和上身以及手的测量值。 这些算法将建模和利用联合统计;即，他们将明确地对手势通道之间的相关性和关联进行建模。 视觉算法还将利用现有注释中的信息，通过监督学习获取模型。 这些算法将允许一个反馈回路，其中在数据分析/聚类期间估计的动态模型可以用于将跟踪模块“调谐”到特定的gestions.It预计，这种方法应该证明在其他HCI应用中分析手势模式是有用的。 作为这项工作的一部分，SignStream系统将与波士顿学院的同事合作，用于严重残疾人基于视觉的界面的用户研究。 视频将通过多个摄像头拍摄，并由HCI研究人员和临床医生进行部分注释，并补充性能数据（速度，准确性，疲劳）和视觉分析算法的输出。更广泛的影响：在这项工作中开发的算法和软件将通过FTP提供给研究社区，作为SignStream的扩展。 因此，这些工具将提供给已经在语言学和计算机人机界面研究中使用SignStream的已建立的、多样化的研究人员。 在这个项目中开发的手势模式分析工具，应加速语言学研究的关键作用，非手动渠道签署的语言和手势交流。 此外，更好地理解非手动和手动通道所起的综合作用，应该会提高基于计算机的手语识别系统以及对视频中观察到的手势成分进行建模的语音识别系统的准确性。 相反，手语和手势交流的合成系统将能够在适当的语言域上建模和生成这些非手动运动，以实现更好的真实感。 最后，希望模式分析算法将有助于更普遍的手势接口的研究。通过这种分析工具获得的见解应有助于改进基于视频的界面系统（例如，对于智力残疾者），其允许更大的舒适性、准确性和易用性。

Research on recognition and generation of signed languages and the gestural component of spoken languages has been hindered by the unavailability of large-scale linguistically annotated corpora of the kind that led to significant advances in the area of spoken language. In addition, the complexity of simultaneous expression of linguistic information on the hands, the face, and the upper body creates special challenges for both linguistic analysis and computer-based recognition. A major goal of this project is the development of pattern analysis algorithms for discovery of the co-occurrence, overlap, relative timing, frequency, and magnitude of linguistically significant movements of the hands, face, and upper body. These will be tested against the PI's recently developed corpus collected from native signers of American Sign Language (ASL).The high-quality video data consist of multiple synchronized movie ?les, showing the signing from multiple angles (including a close-up of the face). Annotations were produced using SignStream (an application developed by the PIs), which enables transcription of parallel streams of information (e.g., movements of the hands, eyes, eyebrows, etc., that convey critical grammatical information in signed languages). The video data and annotations provide a basis for analyzing gestures occurring in the multiple manual and non-manual channels. The goal is to recognize temporal associations both within and across channels. Time-series analysis algorithms will be developed for comparing the degree of similarity of gestural components.Clustering and indexing algorithms will be developed for identification of groups of similar gestural components from mixed discrete annotation event labels (e.g., gloss, eyebrow raise, hand-shape, etc.) and sampled measurement data (e.g., head orientation, direction of eye gaze, hand motion). Since several gestural channels include periodic motions of varying frequency and magnitude (e.g., head nods and shakes), periodicity analysis modules will also be developed. Linguists and computer scientists will collaborate to determine how best to exploit and combine the information available from these different sources. Moreover, the information emerging from the computer science research will be of enormous benefit for the ongoing linguistic research being conducted by the PI's research team on the syntax of ASL. Syntactic research on signed languages has been hindered by the daunting task of attempting to uncover these patterns solely through observation, without the aid of tools for analyzing large amounts of data.To support time-series pattern analysis in SignStream, computer vision algorithms will be developed for analysis of video to extract measurements the head, face, eyes/eyebrows, arms, and upper body, as well as hands. These algorithms will model and exploit joint statistics; i.e., they will explicitly model correlations and associations across gestural channels. The vision algorithms will also make use of information available from the existing annotations to acquire models via supervised learning. These algorithms will allow a feedback loop where the dynamical models estimated during data analysis/clustering can be used to "tune" the tracking modules to specific gestures.It is expected that this approach should prove useful in analyzing gestural patterns in other HCI applications. As part of the effort, the SignStream system will be employed in user studies of vision-based interfaces for the severely handicapped, in collaboration with colleagues at Boston College. Video will be captured via multiple cameras, and partially annotated by the HCI researcher and the clinicians, supplemented with performance data (speed, accuracy, fatigue) and output of the vision analysis algorithms.Broader Impacts: Algorithms and software developed in this effort will be made available to the research community via FTP, as extensions to SignStream. Thus, these tools will be available to the established, diverse group of researchers who already use SignStream in linguistics and computer human interface research. The gestural pattern analysis tools developed in this project should accelerate linguistic research on the critical role of non-manual channels in signed languages and gestural communication. Moreover, better understanding of the combined role that non-manual and manual channels play should lead to improved accuracy in computer-based sign language recognition systems, as well as speech recognition systems that model gestural components observed in video. Conversely, systems for synthesis of signed languages and gestural communication would be able to model and generate these non-manual movements over the appropriate linguistic domains, in order to achieve better realism. Finally, it is hoped that pattern analysis algorithms will be useful in the study of gestural interfaces more generally. Insights gained via such analysis tools should lead to improved video-based interface systems (e.g., for the severely-handicapped) that allow greater comfort, accuracy, and ease of use.