Computational Methods for the Study of American Sign Language Nonmanuals Using Very Large Databases

 描述（由申请人提供）：美国手语（ASL）语法由手动符号（手）和非手动组件（包括面部）指定。我们的一般假设是，非手动面部发音通过比在其他交际系统中看到的更广泛的面部表情集来执行重要的语义和句法功能（例如，言语和情感）。本文将系统地研究这一假说。具体来说，我们将研究以下三个假设，以正确回答上述一般假设：首先，我们假设（H1），在美国手语的小句水平的语法面部表情和/或其激活强度的生产涉及的面部肌肉比在言语和情感中看到的更广泛。其次，我们假设（H2），这些面部结构的时间结构比言语和情感中看到的更广泛。最后，我们假设（H3），从原始视频中删除这些ASL非手动标记，大大降低了正确识别签名句子的子句类型的机会。为了验证这三个假设，我们定义了一个高度创新的方法，基于设计的计算工具的分析非手册签署。我们将特别审查以下三个具体目标。在目标1中，我们将建立一系列计算机算法，使我们能够自动（即，不需要任何人为干预）检测面部、其面部特征以及面部肌肉的运动及其激活强度的自动检测。这些工具将被纳入用于语言分析的标准软件ELAN。然后，这些工具将用于测试六个特定的假设，以成功地研究H1。在目标2中，我们定义了计算机视觉和机器学习算法来识别ASL面部配置的时间结构，并研究这些与语音和情感中所见的时间结构的比较。我们将研究六个具体的假设，以成功地解决H2。在这两个目标中定义了备选假设。最后，在目标3中，我们定义了自动修改ASL中面部表情的原始视频以消除所识别的非手动标记的算法。ASL的本地用户将完成行为实验，以检查H3和测试潜在的替代假设。还将完成与非签名者控制的比较分析。因此，这些研究将进一步验证H1和H2。我们提供证据证明我们有能力成功完成这些目标中的每一项任务。这些目标解决了一个关键的需求;目前，非手册的研究必须手工进行。为了能够得出结论性的结果，有必要研究数千个视频。所提出的计算方法假设至少减少了50倍的时间相比，手工完成的方法。

 DESCRIPTION (provided by applicant): American Sign Language (ASL) grammar is specified by the manual sign (the hands) and by the nonmanual components, which include the face. Our general hypothesis is that nonmanual facial articulations perform significant semantic and syntactic functions by means of a more extensive set of facial expressions than that seen in other communicative systems (e.g., speech and emotion). This proposal will systematically study this hypothesis. Specifically, we will study the following three hypotheses needed to properly answer the general hypothesis stated above: First, we hypothesize (H1) that the facial muscles involved in the production of clause-level grammatical facial expressions in ASL and/or their intensity of activation are more extensive than those seen in speech and emotion. Second, we hypothesize (H2) that the temporal structure of these facial configurations are more extensive than those seen in speech and emotion. Finally, we hypothesize (H3) that eliminating these ASL nonmanual makers from the original videos, drastically reduces the chances of correctly identifying the clause type of the signed sentence. To test these three hypotheses, we define a highly innovative approach based on the design of computational tools for the analysis of nonmanuals in signing. In particular, we will examine the following three specific aims. In Aim 1, we will build a series of computer algorithms that allow us to automatically (i.e., without the need of any human intervention) detect the face, its facial features as well as the automatic detection of the movements of the facial muscles and their intensity of activation. These tools will be integrated into ELAN, a standard software used for linguistic analysis. These tools will then be used to test six specific hypotheses to successfully study H1. In Aim 2, we define computer vision and machine learning algorithms to identify the temporal structure of ASL facial configurations and examine how these compare to those seen in speech and emotion. We will study six specific hypotheses to successfully address H2. Alternative hypotheses are defined in both aims. Finally, in Aim 3 we define algorithms to automatically modify the original videos of facial expression in ASL to eliminate the identified nonmanual markers. Native users of ASL will complete behavioral experiments to examine H3 and test potential alternative hypotheses. Comparative analysis with non-signer controls will also be completed. These studies will thus further validate H1 and H2. We provide evidence of our ability to successfully complete the tasks in each of these aims. These aims address a critical need; at present, the study of nonmanuals must be carried out by hand. To be able to draw conclusive results, it is necessary to study thousands of videos. The proposed computational approach supposes at least a 50-fold reduction in time compared to methods done by hand.