Computational Methods for Analysis of Mouth Shapes in Sign Languages

描述（由申请人提供）：美国手语（ASL）语法由手动符号（手）和非手动组件（脸）指定。这些面部发音执行重要的语义，韵律，语用和句法功能。本文将系统地研究美国手语中的口位。我们的假设是，美国手语嘴的位置是更广泛的比那些用于讲话。为了研究这一假设，本项目分为三个目标。在我们的第一个目标中，我们假设嘴的位置是理解在上下文中产生的符号的基础，因为它们与孤立地看到的符号非常不同。为了研究这一点，我们最近收集了一个数据库的美国手语句子和非手册超过3600个视频剪辑从20个聋人本地签名。我们的实验将使用这个数据库来识别从视觉到语言特征的潜在映射。为了成功地做到这一点，我们的第二个目标是设计一套形状分析和判别分析算法，可以有效地分析这些视频剪辑中的大量帧。我们的目标是定义一个语言上有用的模型，即，包含主要视觉特征的最小模型，可以从中进行进一步的预测。然后，在我们的第三个目标中，我们将探讨语言上不同的嘴部位置在视觉上也不同的假设。特别是，我们将使用第二个目标中定义的算法来确定是否使用不同的视觉特征来定义不同的语言类别。这一结果将表明，是否语言意义的嘴的位置不仅是必要的，在美国手语（假设在目标1），但他们是否被定义使用非重叠的视觉特征（假设在目标3）。这些目标针对的是一个关键需求。目前，非手动的研究必须手动进行，即必须手动记录每个帧中每个面部特征的形状和位置。此外，为了能够得出语言模型设计的结论性结果，有必要研究由不同签名者产生的相关句子的许多视频序列。因此，事实证明，手动继续这项研究几乎是不可能的。在这个过程中设计的算法将有利于本分析的美国手语非手册，并导致更好的教材。 公共卫生相关性：耳聋限制了对信息的获取，从而影响了学术成就，个人融合和终身财务状况，也抑制了聋人对听觉世界的宝贵贡献。我们的研究的公共利益包括：（1）一个实用和有用的设备，以加强聋人和听力正常的人在各种设置之间的沟通的目标;和（2）消除障碍，阻止聋人实现其全部潜力。对非手册的理解也将改变美国手语的教学方式，从而改善对聋人教师、手语翻译和教员以及关键的聋哑儿童父母的培训。

DESCRIPTION (provided by applicant): American Sign Language (ASL) grammar is specified by the manual sign (the hand) and by the nonmanual components (the face). These facial articulations perform significant semantic, prosodic, pragmatic, and syntactic functions. This proposal will systematically study mouth positions in ASL. Our hypothesis is that ASL mouth positions are more extensive than those used in speech. To study this hypothesis, this project is divided into three aims. In our first aim, we hypothesize that mouth positions are fundamental for the understanding of signs produced in context because they are very distinct from signs seen in isolation. To study this we have recently collected a database of ASL sentences and nonmanuals in over 3600 video clips from 20 Deaf native signers. Our experiments will use this database to identify potential mappings from visual to linguistic features. To successfully do this, our second aim is to design a set of shape analysis and discriminant analysis algorithms that can efficiently analyze the large number of frames in these video clips. The goal is to define a linguistically useful model, i.e., the smallest model that contains the main visual features from which further predictions can be made. Then, in our third aim, we will explore the hypothesis that the linguistically distinct mouth positions are also visually distinct. In particular, we will use the algorithms defined in the second aim to determine if distinct visual features are used to define different linguistic categories. This result will show whether linguistically meaningful mouth positions are not only necessary in ASL (as hypothesized in aim 1), but whether they are defined using non-overlapping visual features (as hypothesized in aim 3). These aims address a critical need. At present, the study of nonmanuals must be carried out manually, that is, the shape and position of each facial feature in each frame must be recorded by hand. Furthermore, to be able to draw conclusive results for the design of a linguistic model, it is necessary to study many video sequences of related sentences as produced by different signers. It has thus proven nearly impossible to continue this research manually. The algorithms designed in the course of this grant will facilitate this analysis of ASL nonmanuals and lead to better teaching materials. PUBLIC HEALTH RELEVANCE: Deafness limits access to information, with consequent effects on academic achievement, personal integration, and life-long financial situation, and also inhibits valuable contributions by Deaf people to the hearing world. The public benefit of our research includes: (1) the goal of a practical and useful device to enhance communication between Deaf and hearing people in a variety of settings; and (2) the removal of a barrier that prevents Deaf individuals from achieving their full potential. An understanding of the non-manuals will also change how ASL is taught, leading to an improvement in the training of teachers of the Deaf, sign language interpreters and instructors, and crucially parents of deaf children.