Computational Methods for the Study of American Sign Language Nonmanuals Using Very Large Databases

使用大型数据库研究美国手语非手册的计算方法

• 批准号: 9199411
• 负责人: Aleix M Martinez
• 金额: $ 31.94万
• 依托单位: OHIO STATE UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-01-01 至 2020-12-01
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9199411
• 关键词: Academic achievement; Access to Information; Address; Agreement; Algorithms; American Sign Language; Articulation; Behavioral; Child; Code; Communication; Computational algorithm; Computer Vision Systems; Computer software; Computing Methodologies; Controlled Study; Databases; Detection; Devices; Dimensions; Emotions; Excision; Face; Facial Expression; Facial Muscles; Goals; Hand; Head; Hearing; Hearing Impaired Persons; Human; Image; Individual; Intervention; Life; Linguistics; Logic; Machine Learning; Manuals; Methods; Movement; Parents; Pattern Recognition; Production; Research; Research Personnel; Science; Semantics; Series; Shapes; Sign Language; Signal Transduction; Specific qualifier value; Speech; Structure; System; Teacher Professional Development; Technology; Testing; Time; Visual; Visual system structure; base; body position; comparative; computerized tools; deafness; design; experience; experimental study; face perception; innovation; instructor; interest; prevent; public health relevance; reconstruction; showing emotion; syntax; tool

 DESCRIPTION (provided by applicant): American Sign Language (ASL) grammar is specified by the manual sign (the hands) and by the nonmanual components, which include the face. Our general hypothesis is that nonmanual facial articulations perform significant semantic and syntactic functions by means of a more extensive set of facial expressions than that seen in other communicative systems (e.g., speech and emotion). This proposal will systematically study this hypothesis. Specifically, we will study the following three hypotheses needed to properly answer the general hypothesis stated above: First, we hypothesize (H1) that the facial muscles involved in the production of clause-level grammatical facial expressions in ASL and/or their intensity of activation are more extensive than those seen in speech and emotion. Second, we hypothesize (H2) that the temporal structure of these facial configurations are more extensive than those seen in speech and emotion. Finally, we hypothesize (H3) that eliminating these ASL nonmanual makers from the original videos, drastically reduces the chances of correctly identifying the clause type of the signed sentence. To test these three hypotheses, we define a highly innovative approach based on the design of computational tools for the analysis of nonmanuals in signing. In particular, we will examine the following three specific aims. In Aim 1, we will build a series of computer algorithms that allow us to automatically (i.e., without the need of any human intervention) detect the face, its facial features as well as the automatic detection of the movements of the facial muscles and their intensity of activation. These tools will be integrated into ELAN, a standard software used for linguistic analysis. These tools will then be used to test six specific hypotheses to successfully study H1. In Aim 2, we define computer vision and machine learning algorithms to identify the temporal structure of ASL facial configurations and examine how these compare to those seen in speech and emotion. We will study six specific hypotheses to successfully address H2. Alternative hypotheses are defined in both aims. Finally, in Aim 3 we define algorithms to automatically modify the original videos of facial expression in ASL to eliminate the identified nonmanual markers. Native users of ASL will complete behavioral experiments to examine H3 and test potential alternative hypotheses. Comparative analysis with non-signer controls will also be completed. These studies will thus further validate H1 and H2. We provide evidence of our ability to successfully complete the tasks in each of these aims. These aims address a critical need; at present, the study of nonmanuals must be carried out by hand. To be able to draw conclusive results, it is necessary to study thousands of videos. The proposed computational approach supposes at least a 50-fold reduction in time compared to methods done by hand.

 描述（由申请人提供）：美国手语 (ASL) 语法由手动手势（手）和非手动组件（包括面部）指定。我们的一般假设是，非手动面部清晰度通过比其他交流系统（例如言语和情感）更广泛的面部表情来执行重要的语义和句法功能。本提案将系统地研究这一假设。具体来说，我们将研究正确回答上述一般假设所需的以下三个假设：首先，我们假设（H1）美国手语中参与产生子句级语法面部表情的面部肌肉和/或其激活强度比言语和情感中所见的面部肌肉更广泛。其次，我们假设（H2）这些面部配置的时间结构比言语和情感中看到的时间结构更广泛。最后，我们假设（H3）从原始视频中消除这些 ASL 非手动制作者，会大大降低正确识别签名句子的子句类型的机会。为了检验这三个假设，我们基于计算工具的设计定义了一种高度创新的方法，用于分析签名中的非手册。我们将特别研究以下三个具体目标。在目标 1 中，我们将构建一系列计算机算法，使我们能够自动（即无需任何人为干预）检测面部及其面部特征，并自动检测面部肌肉的运动及其激活强度。这些工具将集成到用于语言分析的标准软件 ELAN 中。然后，这些工具将用于测试六个具体假设，以成功研究 H1。在目标 2 中，我们定义了计算机视觉和机器学习算法来识别 ASL 面部配置的时间结构，并检查它们与语音和情感中看到的配置的比较。我们将研究六个具体假设以成功解决 H2。两个目标都定义了替代假设。最后，在目标 3 中，我们定义了算法来自动修改 ASL 中面部表情的原始视频，以消除已识别的非手动标记。 ASL 的本地用户将完成行为实验来检查 H3 并测试潜在的替代假设。与非签名者控制的比较分析也将完成。因此，这些研究将进一步验证 H1 和 H2。我们提供证据证明我们有能力成功完成每个目标的任务。这些目标满足了关键需求；目前，非手册的学习必须靠手工进行。为了能够得出结论性的结果，有必要研究数千个视频。与手工方法相比，所提出的计算方法假设时间至少减少 50 倍。