Computational Methods for the Study of American Sign Language Nonmanuals Using Very Large Databases

 描述(由申请人提供)：美国手语(ASL)语法由手势(手)和非手势成分(包括面部)指定。我们的一般假设是，与其他交际系统(如语音和情感)相比，非手动面部发音通过更广泛的面部表情集来执行重要的语义和句法功能。这项提议将系统地研究这一假说。具体地说，我们将研究以下三个假设来恰当地回答上面所述的一般假设：第一，我们假设(H1)在ASL中参与产生小句级语法面部表达的面部肌肉和/或它们的激活强度比在言语和情感中看到的更广泛。其次，我们假设(H2)这些面部配置的时间结构比语音和情感中看到的更广泛。最后，我们假设(H3)从原始视频中消除这些ASL非手动标记，大大降低了正确识别签名句子的从句类型的机会。为了检验这三个假设，我们定义了一种基于设计计算工具的高度创新的方法，用于分析签名中的非手册。特别是，我们将审查以下三个具体目标。在目标1中，我们将建立一系列计算机算法，使我们能够自动(即，无需任何人工干预)检测面部及其面部特征，以及自动检测面部肌肉的运动及其激活强度。这些工具将集成到用于语言分析的标准软件ELAN中。然后，这些工具将用于测试六个特定的假设，以成功研究H1。在目标2中，我们定义了计算机视觉和机器学习算法来识别ASL面部配置的时间结构，并检查这些与语音和情感中看到的相比如何。我们将研究六个具体的假设来成功地解决H2。在两个目标中都定义了可供选择的假设。最后，在目标3中，我们定义了自动修改ASL中原始人脸表情视频的算法，以消除识别出的非人工标记。ASL的本地用户将完成行为实验，以检查H3并测试潜在的替代假设。还将完成与非签字人对照的比较分析。因此，这些研究将进一步验证H1和H2。我们提供证据，证明我们有能力成功完成这些目标中的每一项任务。这些目标解决了一个关键的需求；目前，非手册的学习必须手工进行。为了能够得出决定性的结果，有必要研究数千个视频。建议的计算方法假设与手工方法相比，时间至少减少了50倍。

 DESCRIPTION (provided by applicant): American Sign Language (ASL) grammar is specified by the manual sign (the hands) and by the nonmanual components, which include the face. Our general hypothesis is that nonmanual facial articulations perform significant semantic and syntactic functions by means of a more extensive set of facial expressions than that seen in other communicative systems (e.g., speech and emotion). This proposal will systematically study this hypothesis. Specifically, we will study the following three hypotheses needed to properly answer the general hypothesis stated above: First, we hypothesize (H1) that the facial muscles involved in the production of clause-level grammatical facial expressions in ASL and/or their intensity of activation are more extensive than those seen in speech and emotion. Second, we hypothesize (H2) that the temporal structure of these facial configurations are more extensive than those seen in speech and emotion. Finally, we hypothesize (H3) that eliminating these ASL nonmanual makers from the original videos, drastically reduces the chances of correctly identifying the clause type of the signed sentence. To test these three hypotheses, we define a highly innovative approach based on the design of computational tools for the analysis of nonmanuals in signing. In particular, we will examine the following three specific aims. In Aim 1, we will build a series of computer algorithms that allow us to automatically (i.e., without the need of any human intervention) detect the face, its facial features as well as the automatic detection of the movements of the facial muscles and their intensity of activation. These tools will be integrated into ELAN, a standard software used for linguistic analysis. These tools will then be used to test six specific hypotheses to successfully study H1. In Aim 2, we define computer vision and machine learning algorithms to identify the temporal structure of ASL facial configurations and examine how these compare to those seen in speech and emotion. We will study six specific hypotheses to successfully address H2. Alternative hypotheses are defined in both aims. Finally, in Aim 3 we define algorithms to automatically modify the original videos of facial expression in ASL to eliminate the identified nonmanual markers. Native users of ASL will complete behavioral experiments to examine H3 and test potential alternative hypotheses. Comparative analysis with non-signer controls will also be completed. These studies will thus further validate H1 and H2. We provide evidence of our ability to successfully complete the tasks in each of these aims. These aims address a critical need; at present, the study of nonmanuals must be carried out by hand. To be able to draw conclusive results, it is necessary to study thousands of videos. The proposed computational approach supposes at least a 50-fold reduction in time compared to methods done by hand.