HCC: Medium: Collaborative Research: Generating Accurate, Understandable Sign Language Animations Based on Analysis of Human Signing

美国手语（ASL）动画有可能使美国许多只具备有限英语读写能力的聋人成年人能够获得信息。在这项涉及三个机构合作的研究中，PI的目标是通过计算技术更好地了解ASL语言学，同时推进ASL动画生成的最新技术，以供聋人使用。 为此，PI将开发两个方面的ASL生产语言为基础的模型：头部姿势和面部表情所需的运动，携带必要的语法信息，并经常扩展到大于一个单一的标志，以及ASL签名的手动和非手动元素的时间和协调。 初步工作表明，这些问题显着影响签名者如何理解ASL动画，这些方面的当前ASL动画技术需要改进。 人类或动画角色的脸应该如何表达，才能准确地表现出语言学上有意义的面部表情，这些面部表情是美国手语语法的一部分？ 这些运动的起始、偏移和过渡应该如何产生？ 如何在时间上协调面部表情和手部动作，使美国手语的表达尽可能地符合语法要求和易于理解？ 为了回答这些悬而未决的问题，PI的新方法将把计算机视觉技术应用于从人类签名者那里收集的带有语言注释的视频数据，以便生成用于动画制作的模型。PI将通过新的数据收集和注释扩展其现有的注释视频ASL语料库，并将分析这些数据，以研究ASL制作的手动和非手动组件的使用，定时和同步。 这些带注释的视频将用于训练高质量的计算机视觉模型，以识别具有语言意义的面部表情和时间微妙性。 这些计算机视觉模型的参数将用于假设ASL定时和面部运动的计算模型，将其纳入ASL动画生成软件并由本地签名者进行评估。 这些模型将在基于用户的研究周期中反复完善，并融入ASL动画技术，以更准确地模仿人类手势。 项目成果将包括用于ASL表演动画的虚拟人物运动的高质量模型。 ASL的视频语料库的分析将产生新的语言学见解的微观面部表情和时间协调的脸和手在ASL生产，而在ASL韵律分析的进步将有助于理解的基本共性和特定模态之间的差异签署和口头语言，这是必不可少的人类语言能力的充分理解。新的建模方法和识别技术的建立将促进计算机视觉领域的发展，有利于在ASL快速复杂运动期间识别和跟踪视频中的人脸和身体（以及其他形式的人类流动）。这项研究将导致技术的重大改进，用于生成语言准确的ASL动画，这将使信息，应用程序，网站和服务更容易为大量英语读写能力相对较低的聋人所利用。 用于识别人类视频中的ASL的计算机视觉技术的进步将在人机交互、面部表情的识别和动画以及计算机视觉中具有普遍适用性。 在这个项目中创建的语料库将使语言学和计算机科学的学生和研究人员（包括那些无法获得必要的技术和人力资源来从本地签名者和时间密集的语言注释中进行自己的数据收集的学生和研究人员）能够从事对美国手语的研究。 待开发的技术也将使部分自动化的耗时的创建注释ASL视频语料库。 与PI早期的工作一样，拟议的研究将为聋人和其他代表性不足的群体成员创造参与科学研究的机会。

American Sign Language (ASL) animations have the potential to make information accessible to many deaf adults in the United States who possess only limited English literacy. In this research, which involves collaboration across three institutions, the PIs' goal is to gain a better understanding of ASL linguistics through computational techniques while advancing the state of the art in the generation of ASL animations for accessibility applications for people who are deaf. To these ends, the PIs will develop linguistically based models of two aspects of ASL production: movements required for head gestures and facial expressions that carry essential grammatical information and frequently extend over domains larger than a single sign, and the timing and coordination of manual and non-manual elements of ASL signing. Preliminary work has shown that these issues significantly affect how well signers understand ASL animations, and that these aspects of current ASL animation technologies require improvement. How should the face of a human or animated character be articulated to perform, with accuracy, the linguistically meaningful facial expressions that are part of ASL grammar? How should the onsets, offsets, and transitions of these movements be produced? How should the facial expressions and hand movements be temporally coordinated so that the ASL production is as grammatically correct and understandable as possible? To answer open questions such as these, the PIs' novel approach will apply techniques from computer vision to linguistically annotated video data collected from human signers, in order to produce models for use in animation-production. The PIs will expand their existing annotated video ASL corpora through new data collection and annotation, and will analyze these data to study the use, timing, and synchronization of manual and non-manual components of ASL production. The annotated videos will be used to train high quality computer vision models for recognition of linguistically significant facial expressions and timing subtleties. Parameters of these computer vision models will be used to hypothesize computational models of ASL timing and facial movements, to be incorporated into ASL-animation generation software and evaluated by native signers. The models will be iteratively refined in cycles of user-based studies and incorporated into ASL animation technologies to more accurately mimic human signing. Project outcomes will include high quality models of the movement of virtual human characters for animations of ASL performance. The analysis of video corpora of ASL will produce new linguistic insights into the micro-facial expressions and the temporal coordination of the face and hands in ASL production, while advances in the analysis of ASL prosody will contribute to an understanding of the fundamental commonalities and modality-specific differences between signed and spoken languages that is essential to a full understanding of the human language faculty. The creation of new modeling approaches and recognition techniques will advance the field of computer vision, by benefiting the identification and tracking of the human face and body in video during the rapid and complex movements of ASL (and other forms of human movement).Broader Impacts: This research will lead to significant improvements to technology for generating linguistically accurate ASL animations, which will make information, applications, websites, and services more accessible to the large number of deaf individuals with relatively low English literacy. Advances in computer vision techniques for recognizing ASL in videos of humans will have general applicability in human-computer interaction, recognition and animation of facial expressions, and computer vision. The corpora created in this project will enable students and researchers in both linguistics and computer science (including those without access to the requisite technological and human resources to carry out their own data collection from native signers and time-intensive linguistic annotations) to engage in research on ASL. The techniques to be developed will also enable partial automation of the time-consuming creation of annotated ASL video corpora. As in the PIs' earlier work, the proposed research will create opportunities for people who are deaf and members of other underrepresented groups to participate in scientific research.