Computational Methods for Analysis of Mouth Shapes in Sign Languages

描述(申请人提供)：美国手语(ASL)语法由手势(手)和非手势成分(脸)指定。这些面部发音具有重要的语义、韵律、语用和句法功能。这项建议将系统地研究ASL患者的口腔位置。我们的假设是，自闭症患者的嘴部位置比说话时所用的位置更广泛。为了研究这一假设，这个项目分为三个目标。在我们的第一个目标中，我们假设嘴部位置是理解上下文中产生的符号的基础，因为它们与孤立地看到的符号非常不同。为了研究这一点，我们最近收集了来自20名聋人手语者的3600多个视频剪辑中的ASL句子和非手册的数据库。我们的实验将使用这个数据库来识别从视觉到语言特征的潜在映射。为了成功地做到这一点，我们的第二个目标是设计一套形状分析和判别分析算法，能够有效地分析这些视频片段中的大量帧。我们的目标是定义一个在语言上有用的模型，即包含主要视觉特征的最小模型，根据这些特征可以做出进一步的预测。然后，在我们的第三个目标中，我们将探索这样一个假设，即语言上不同的嘴巴位置在视觉上也是不同的。特别是，我们将使用第二个目标中定义的算法来确定是否使用不同的视觉特征来定义不同的语言类别。这一结果将表明，在ASL中，语言上有意义的嘴部位置是否不仅是必要的(如目标1中的假设)，而且是否使用非重叠的视觉特征来定义(如目标3中的假设)。这些目标解决了一个紧迫的需求。目前，对非手册的学习必须手动进行，即每一帧中每个面部特征的形状和位置都必须由人工记录。此外，为了能够为语言模型的设计得出决定性的结果，有必要研究由不同签名者制作的大量相关句子的视频序列。因此，事实证明，手动继续这项研究几乎是不可能的。在这项资助过程中设计的算法将促进对美国手语非手册的分析，并导致更好的教材。 公共卫生相关性：耳聋限制了信息的获取，从而影响了学业成就、个人融合和终身经济状况，还抑制了聋人对听力世界的宝贵贡献。我们研究的公共利益包括：(1)目标是开发一种实用和有用的设备，在各种环境下加强聋人和听力人之间的交流；以及(2)消除阻碍聋人充分发挥其潜力的障碍。对非手册的理解也将改变美国手语的教学方式，导致聋人教师、手语翻译和指导员以及关键是聋童父母的培训得到改善。

DESCRIPTION (provided by applicant): American Sign Language (ASL) grammar is specified by the manual sign (the hand) and by the nonmanual components (the face). These facial articulations perform significant semantic, prosodic, pragmatic, and syntactic functions. This proposal will systematically study mouth positions in ASL. Our hypothesis is that ASL mouth positions are more extensive than those used in speech. To study this hypothesis, this project is divided into three aims. In our first aim, we hypothesize that mouth positions are fundamental for the understanding of signs produced in context because they are very distinct from signs seen in isolation. To study this we have recently collected a database of ASL sentences and nonmanuals in over 3600 video clips from 20 Deaf native signers. Our experiments will use this database to identify potential mappings from visual to linguistic features. To successfully do this, our second aim is to design a set of shape analysis and discriminant analysis algorithms that can efficiently analyze the large number of frames in these video clips. The goal is to define a linguistically useful model, i.e., the smallest model that contains the main visual features from which further predictions can be made. Then, in our third aim, we will explore the hypothesis that the linguistically distinct mouth positions are also visually distinct. In particular, we will use the algorithms defined in the second aim to determine if distinct visual features are used to define different linguistic categories. This result will show whether linguistically meaningful mouth positions are not only necessary in ASL (as hypothesized in aim 1), but whether they are defined using non-overlapping visual features (as hypothesized in aim 3). These aims address a critical need. At present, the study of nonmanuals must be carried out manually, that is, the shape and position of each facial feature in each frame must be recorded by hand. Furthermore, to be able to draw conclusive results for the design of a linguistic model, it is necessary to study many video sequences of related sentences as produced by different signers. It has thus proven nearly impossible to continue this research manually. The algorithms designed in the course of this grant will facilitate this analysis of ASL nonmanuals and lead to better teaching materials. PUBLIC HEALTH RELEVANCE: Deafness limits access to information, with consequent effects on academic achievement, personal integration, and life-long financial situation, and also inhibits valuable contributions by Deaf people to the hearing world. The public benefit of our research includes: (1) the goal of a practical and useful device to enhance communication between Deaf and hearing people in a variety of settings; and (2) the removal of a barrier that prevents Deaf individuals from achieving their full potential. An understanding of the non-manuals will also change how ASL is taught, leading to an improvement in the training of teachers of the Deaf, sign language interpreters and instructors, and crucially parents of deaf children.