Computational Methods for the Study of American Sign Language Nonmanuals Using Very Large Databases

使用大型数据库研究美国手语非手册的计算方法

• 批准号: 9054574
• 负责人: Aleix M Martinez
• 金额: $ 33.13万
• 依托单位: OHIO STATE UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-01-01 至 2020-12-01
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9054574
• 关键词: Academic achievement; Access to Information; Address; Agreement; Algorithms; Behavioral; Child; Code; Communication; Computational algorithm; Computer Vision Systems; Computer software; Computing Methodologies; Controlled Study; Databases; Detection; Devices; Educational process of instructing; Emotions; Excision; Face; Facial Expression; Facial Muscles; Goals; Hand; Head; Hearing; Hearing Impaired Persons; Human; Image; Individual; Intervention; Joints; Life; Linguistics; Logic; Machine Learning; Manuals; Methods; Movement; Parents; Pattern Recognition; Production; Research; Research Personnel; Science; Semantics; Series; Shapes; Sign Language; Signal Transduction; Specific qualifier value; Speech; Structure; System; Teacher Professional Development; Technology; Testing; Time; Visual; Visual system structure; base; body position; comparative; computerized tools; deafness; design; experience; face perception; innovation; instructor; interest; prevent; public health relevance; reconstruction; research study; showing emotion; syntax; tool

 DESCRIPTION (provided by applicant): American Sign Language (ASL) grammar is specified by the manual sign (the hands) and by the nonmanual components, which include the face. Our general hypothesis is that nonmanual facial articulations perform significant semantic and syntactic functions by means of a more extensive set of facial expressions than that seen in other communicative systems (e.g., speech and emotion). This proposal will systematically study this hypothesis. Specifically, we will study the following three hypotheses needed to properly answer the general hypothesis stated above: First, we hypothesize (H1) that the facial muscles involved in the production of clause-level grammatical facial expressions in ASL and/or their intensity of activation are more extensive than those seen in speech and emotion. Second, we hypothesize (H2) that the temporal structure of these facial configurations are more extensive than those seen in speech and emotion. Finally, we hypothesize (H3) that eliminating these ASL nonmanual makers from the original videos, drastically reduces the chances of correctly identifying the clause type of the signed sentence. To test these three hypotheses, we define a highly innovative approach based on the design of computational tools for the analysis of nonmanuals in signing. In particular, we will examine the following three specific aims. In Aim 1, we will build a series of computer algorithms that allow us to automatically (i.e., without the need of any human intervention) detect the face, its facial features as well as the automatic detection of the movements of the facial muscles and their intensity of activation. These tools will be integrated into ELAN, a standard software used for linguistic analysis. These tools will then be used to test six specific hypotheses to successfully study H1. In Aim 2, we define computer vision and machine learning algorithms to identify the temporal structure of ASL facial configurations and examine how these compare to those seen in speech and emotion. We will study six specific hypotheses to successfully address H2. Alternative hypotheses are defined in both aims. Finally, in Aim 3 we define algorithms to automatically modify the original videos of facial expression in ASL to eliminate the identified nonmanual markers. Native users of ASL will complete behavioral experiments to examine H3 and test potential alternative hypotheses. Comparative analysis with non-signer controls will also be completed. These studies will thus further validate H1 and H2. We provide evidence of our ability to successfully complete the tasks in each of these aims. These aims address a critical need; at present, the study of nonmanuals must be carried out by hand. To be able to draw conclusive results, it is necessary to study thousands of videos. The proposed computational approach supposes at least a 50-fold reduction in time compared to methods done by hand.

 描述（由应用程序提供）：美国手语（ASL）语法由手动标志（手）和包括面部的非手动组件指定。我们的总体假设是，非人工面部表述通过比其他交流系统（例如言语和情感）更广泛的面部表达来执行重要的语义和句法功能。该建议将系统地研究这一假设。具体而言，我们将研究正确回答上述一般假设所需的以下三个假设：首先，我们假设（H1），在ASL和/或它们的激活强度中，参与条款级别的语法面部表情所涉及的面部肌肉比在语音和情感中所看到的更广泛。其次，我们假设（H2）这些面部配置的临时结构比语音和情感中看到的更广泛。最后，我们假设（H3）从原始视频中消除了这些ASL非手动制造商，从而大大减少了正确识别签名句子的子句类型的机会。为了检验这三个假设，我们根据计算工具的设计来定义一种高度创新的方法，用于分析签名中的非手术。特别是，我们将研究以下三个特定目标。在AIM 1中，我们将构建一系列计算机算法，使我们能够自动（即无需任何人类干预）自动检测面部，其面部特征以及对面部肌肉运动的自动检测及其激活强度。这些工具将集成到Elan，这是一种用于语言分析的标准软件。然后，这些工具将用于检验六个特定假设以成功研究H1。在AIM 2中，我们定义了计算机视觉和机器学习算法，以识别ASL面部配置的临时结构，并研究这些算法与语音和情感中看到的算法如何比较。我们将研究六个特定的假设，以成功解决H2。两个目的都定义了替代假设。最后，在AIM 3中，我们定义算法以自动修改ASL中面部表达的原始视频，以消除已识别的非手动标记。 ASL的本地用户将完成行为实验，以检查H3并检验潜在的替代假设。与非签名对照组的比较分析也将完成。因此，这些研究将进一步验证H1和H2。我们提供了我们能够成功完成这些目标中任务的能力的证据。这些目的满足了一个迫切的需求；目前，必须手工进行非手术的研究。为了能够得出结论性的结果，有必要学习数千个视频。与手工完成的方法相比，提出的计算方法至少假定时间至少减少了50倍。