Pattern Discovery in Signed Languages and Gestural Communication

手语和手势交流中的模式发现

• 批准号: 0329009
• 负责人: Carol Neidle
• 金额: $ 75万
• 依托单位: Trustees of Boston University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2003
• 资助国家: 美国
• 起止时间: 2003-09-15 至 2007-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0329009&HistoricalAwards=false
• 关键词: Pattern; Discovery; Signed; Languages; Gestural

Research on recognition and generation of signed languages and the gestural component of spoken languages has been hindered by the unavailability of large-scale linguistically annotated corpora of the kind that led to significant advances in the area of spoken language. In addition, the complexity of simultaneous expression of linguistic information on the hands, the face, and the upper body creates special challenges for both linguistic analysis and computer-based recognition. A major goal of this project is the development of pattern analysis algorithms for discovery of the co-occurrence, overlap, relative timing, frequency, and magnitude of linguistically significant movements of the hands, face, and upper body. These will be tested against the PI's recently developed corpus collected from native signers of American Sign Language (ASL).The high-quality video data consist of multiple synchronized movie ?les, showing the signing from multiple angles (including a close-up of the face). Annotations were produced using SignStream (an application developed by the PIs), which enables transcription of parallel streams of information (e.g., movements of the hands, eyes, eyebrows, etc., that convey critical grammatical information in signed languages). The video data and annotations provide a basis for analyzing gestures occurring in the multiple manual and non-manual channels. The goal is to recognize temporal associations both within and across channels. Time-series analysis algorithms will be developed for comparing the degree of similarity of gestural components.Clustering and indexing algorithms will be developed for identification of groups of similar gestural components from mixed discrete annotation event labels (e.g., gloss, eyebrow raise, hand-shape, etc.) and sampled measurement data (e.g., head orientation, direction of eye gaze, hand motion). Since several gestural channels include periodic motions of varying frequency and magnitude (e.g., head nods and shakes), periodicity analysis modules will also be developed. Linguists and computer scientists will collaborate to determine how best to exploit and combine the information available from these different sources. Moreover, the information emerging from the computer science research will be of enormous benefit for the ongoing linguistic research being conducted by the PI's research team on the syntax of ASL. Syntactic research on signed languages has been hindered by the daunting task of attempting to uncover these patterns solely through observation, without the aid of tools for analyzing large amounts of data.To support time-series pattern analysis in SignStream, computer vision algorithms will be developed for analysis of video to extract measurements the head, face, eyes/eyebrows, arms, and upper body, as well as hands. These algorithms will model and exploit joint statistics; i.e., they will explicitly model correlations and associations across gestural channels. The vision algorithms will also make use of information available from the existing annotations to acquire models via supervised learning. These algorithms will allow a feedback loop where the dynamical models estimated during data analysis/clustering can be used to "tune" the tracking modules to specific gestures.It is expected that this approach should prove useful in analyzing gestural patterns in other HCI applications. As part of the effort, the SignStream system will be employed in user studies of vision-based interfaces for the severely handicapped, in collaboration with colleagues at Boston College. Video will be captured via multiple cameras, and partially annotated by the HCI researcher and the clinicians, supplemented with performance data (speed, accuracy, fatigue) and output of the vision analysis algorithms.Broader Impacts: Algorithms and software developed in this effort will be made available to the research community via FTP, as extensions to SignStream. Thus, these tools will be available to the established, diverse group of researchers who already use SignStream in linguistics and computer human interface research. The gestural pattern analysis tools developed in this project should accelerate linguistic research on the critical role of non-manual channels in signed languages and gestural communication. Moreover, better understanding of the combined role that non-manual and manual channels play should lead to improved accuracy in computer-based sign language recognition systems, as well as speech recognition systems that model gestural components observed in video. Conversely, systems for synthesis of signed languages and gestural communication would be able to model and generate these non-manual movements over the appropriate linguistic domains, in order to achieve better realism. Finally, it is hoped that pattern analysis algorithms will be useful in the study of gestural interfaces more generally. Insights gained via such analysis tools should lead to improved video-based interface systems (e.g., for the severely-handicapped) that allow greater comfort, accuracy, and ease of use.

由于缺乏大规模的语言注释语料库，手语和口语手势成分的识别和生成研究一直受到阻碍，而这种语料库导致了口语领域的重大进展。此外，语言信息在手、脸和上半身同时表达的复杂性给语言分析和基于计算机的识别带来了特殊的挑战。该项目的一个主要目标是开发模式分析算法，用于发现手、脸和上半身在语言上的重要运动的共现、重叠、相对时间、频率和幅度。这些将与PI最近开发的从美国手语（ASL）的本地签字人收集的语料库进行测试。高质量的视频数据由多个同步电影组成。从多个角度（包括脸部特写）展示签名。注释是使用SignStream（由pi开发的应用程序）生成的，它可以转录平行信息流（例如，手、眼睛、眉毛等的动作，这些动作在手语中传达了关键的语法信息）。视频数据和注释为分析多个手动和非手动通道中发生的手势提供了基础。目标是识别通道内和通道间的时间关联。将开发时间序列分析算法来比较手势成分的相似程度。将开发聚类和索引算法，用于从混合离散注释事件标签（例如，光泽，眉毛扬起，手部形状等）和采样测量数据（例如，头部方向，眼睛注视方向，手部运动）中识别相似手势成分组。由于几个手势通道包括不同频率和大小的周期性运动（例如，点头和摇晃），周期性分析模块也将被开发。语言学家和计算机科学家将合作确定如何最好地利用和整合来自这些不同来源的信息。此外，从计算机科学研究中获得的信息将对PI研究小组正在进行的美国手语语法语言学研究产生巨大的好处。在没有分析大量数据的工具的帮助下，试图仅通过观察来揭示这些模式的艰巨任务阻碍了手语语法研究。为了支持SignStream中的时间序列模式分析，将开发用于分析视频的计算机视觉算法，以提取头部、面部、眼睛/眉毛、手臂、上半身以及手部的测量值。这些算法将建模和利用联合统计；也就是说，他们将明确地模拟跨手势通道的相关性和关联。视觉算法还将利用现有注释中的可用信息，通过监督学习获得模型。这些算法将允许一个反馈循环，其中在数据分析/聚类期间估计的动态模型可用于“调整”跟踪模块以适应特定的手势。期望这种方法在分析其他HCI应用中的手势模式方面证明是有用的。作为这项工作的一部分，SignStream系统将与波士顿学院的同事合作，用于重度残疾人基于视觉的界面的用户研究。视频将通过多个摄像头捕获，由HCI研究人员和临床医生进行部分注释，并辅以性能数据（速度、准确性、疲劳程度）和视觉分析算法的输出。更广泛的影响：在这项工作中开发的算法和软件将通过FTP作为SignStream的扩展提供给研究社区。因此，这些工具将提供给已经在语言学和计算机人机界面研究中使用SignStream的研究人员。本项目开发的手势模式分析工具将加速非手动通道在手语和手势交流中的关键作用的语言学研究。此外，更好地理解非手动和手动通道的综合作用，应该会提高基于计算机的手语识别系统的准确性，以及模拟视频中观察到的手势成分的语音识别系统。相反，合成手语和手势交流的系统将能够在适当的语言领域建模和生成这些非手动动作，以实现更好的真实感。最后，希望模式分析算法能在手势界面的研究中发挥更大的作用。通过这种分析工具获得的见解将导致改进基于视频的界面系统（例如，为严重残疾的人），从而提供更大的舒适性、准确性和易用性。