Computational Methods for Analysis of Mouth Shapes in Sign Languages

点击翻译按钮获取中文摘要

DESCRIPTION (provided by applicant): American Sign Language (ASL) grammar is specified by the manual sign (the hand) and by the nonmanual components (the face). These facial articulations perform significant semantic, prosodic, pragmatic, and syntactic functions. This proposal will systematically study mouth positions in ASL. Our hypothesis is that ASL mouth positions are more extensive than those used in speech. To study this hypothesis, this project is divided into three aims. In our first aim, we hypothesize that mouth positions are fundamental for the understanding of signs produced in context because they are very distinct from signs seen in isolation. To study this we have recently collected a database of ASL sentences and nonmanuals in over 3600 video clips from 20 Deaf native signers. Our experiments will use this database to identify potential mappings from visual to linguistic features. To successfully do this, our second aim is to design a set of shape analysis and discriminant analysis algorithms that can efficiently analyze the large number of frames in these video clips. The goal is to define a linguistically useful model, i.e., the smallest model that contains the main visual features from which further predictions can be made. Then, in our third aim, we will explore the hypothesis that the linguistically distinct mouth positions are also visually distinct. In particular, we will use the algorithms defined in the second aim to determine if distinct visual features are used to define different linguistic categories. This result will show whether linguistically meaningful mouth positions are not only necessary in ASL (as hypothesized in aim 1), but whether they are defined using non-overlapping visual features (as hypothesized in aim 3). These aims address a critical need. At present, the study of nonmanuals must be carried out manually, that is, the shape and position of each facial feature in each frame must be recorded by hand. Furthermore, to be able to draw conclusive results for the design of a linguistic model, it is necessary to study many video sequences of related sentences as produced by different signers. It has thus proven nearly impossible to continue this research manually. The algorithms designed in the course of this grant will facilitate this analysis of ASL nonmanuals and lead to better teaching materials. PUBLIC HEALTH RELEVANCE: Deafness limits access to information, with consequent effects on academic achievement, personal integration, and life-long financial situation, and also inhibits valuable contributions by Deaf people to the hearing world. The public benefit of our research includes: (1) the goal of a practical and useful device to enhance communication between Deaf and hearing people in a variety of settings; and (2) the removal of a barrier that prevents Deaf individuals from achieving their full potential. An understanding of the non-manuals will also change how ASL is taught, leading to an improvement in the training of teachers of the Deaf, sign language interpreters and instructors, and crucially parents of deaf children.