Computational Methods for Analysis of Mouth Shapes in Sign Languages

描述（由申请人提供）：美国手语（ASL）语法由手势（手）和非手动成分（脸）指定。这些面部发音具有重要的语义、韵律、语用和句法功能。本提案将系统地研究美国手语中的嘴位。我们的假设是，美国手语的嘴位比口语中使用的嘴位更广泛。为了研究这一假设，本项目分为三个目标。在我们的第一个目标中，我们假设嘴位是理解语境中产生的符号的基础，因为它们与孤立的符号非常不同。为了研究这一点，我们最近收集了一个数据库，其中包括来自20名聋人母语手语的3600多个视频片段中的美国手语句子和非手册。我们的实验将使用这个数据库来识别从视觉特征到语言特征的潜在映射。为了成功地做到这一点，我们的第二个目标是设计一套形状分析和判别分析算法，可以有效地分析这些视频片段中的大量帧。目标是定义一个语言上有用的模型，即包含主要视觉特征的最小模型，从而可以进行进一步的预测。然后，在我们的第三个目标中，我们将探索一个假设，即语言上不同的嘴位在视觉上也是不同的。特别是，我们将使用第二个目标中定义的算法来确定是否使用不同的视觉特征来定义不同的语言类别。这一结果将表明，在语言上有意义的嘴位是否不仅在美国手语中是必要的（如目标1中的假设），而且它们是否使用非重叠的视觉特征来定义（如目标3中的假设）。这些目标解决了一项关键需求。目前，非手册的研究必须手工进行，即必须手工记录每一帧中每个面部特征的形状和位置。此外，为了能够为语言模型的设计得出结论性的结果，有必要研究由不同的手语者产生的相关句子的许多视频序列。因此，人工继续进行这项研究几乎是不可能的。本项目中设计的算法将有助于对非手册类美国手语的分析，并产生更好的教学材料。

DESCRIPTION (provided by applicant): American Sign Language (ASL) grammar is specified by the manual sign (the hand) and by the nonmanual components (the face). These facial articulations perform significant semantic, prosodic, pragmatic, and syntactic functions. This proposal will systematically study mouth positions in ASL. Our hypothesis is that ASL mouth positions are more extensive than those used in speech. To study this hypothesis, this project is divided into three aims. In our first aim, we hypothesize that mouth positions are fundamental for the understanding of signs produced in context because they are very distinct from signs seen in isolation. To study this we have recently collected a database of ASL sentences and nonmanuals in over 3600 video clips from 20 Deaf native signers. Our experiments will use this database to identify potential mappings from visual to linguistic features. To successfully do this, our second aim is to design a set of shape analysis and discriminant analysis algorithms that can efficiently analyze the large number of frames in these video clips. The goal is to define a linguistically useful model, i.e., the smallest model that contains the main visual features from which further predictions can be made. Then, in our third aim, we will explore the hypothesis that the linguistically distinct mouth positions are also visually distinct. In particular, we will use the algorithms defined in the second aim to determine if distinct visual features are used to define different linguistic categories. This result will show whether linguistically meaningful mouth positions are not only necessary in ASL (as hypothesized in aim 1), but whether they are defined using non-overlapping visual features (as hypothesized in aim 3). These aims address a critical need. At present, the study of nonmanuals must be carried out manually, that is, the shape and position of each facial feature in each frame must be recorded by hand. Furthermore, to be able to draw conclusive results for the design of a linguistic model, it is necessary to study many video sequences of related sentences as produced by different signers. It has thus proven nearly impossible to continue this research manually. The algorithms designed in the course of this grant will facilitate this analysis of ASL nonmanuals and lead to better teaching materials. PUBLIC HEALTH RELEVANCE: Deafness limits access to information, with consequent effects on academic achievement, personal integration, and life-long financial situation, and also inhibits valuable contributions by Deaf people to the hearing world. The public benefit of our research includes: (1) the goal of a practical and useful device to enhance communication between Deaf and hearing people in a variety of settings; and (2) the removal of a barrier that prevents Deaf individuals from achieving their full potential. An understanding of the non-manuals will also change how ASL is taught, leading to an improvement in the training of teachers of the Deaf, sign language interpreters and instructors, and crucially parents of deaf children.