CAREER: Learning to Generate American Sign Language Animation through Motion-Capture and Participation of Native ASL Signers

美国手语（ASL）是美国约50万人的主要交流方式。美国手语是一种不同于英语的语言；事实上，大多数美国聋人高中毕业生的英语阅读水平只有四年级（10岁）。因此，许多聋哑人发现很难在电脑上、电视字幕或其他环境中阅读英语文本。将英语文本翻译成人类角色表演美国手语的动画的软件将为聋哑人提供更多的信息和服务。然而，不幸的是，现代计算语言软件还没有对美国手语的基本方面进行建模。具体来说，手语使用者会将讨论的实体与他们身体周围的3D位置联系起来，并且许多类型的手语动作会根据这些位置发生变化：代词、限定词、许多名词短语、许多类型的动词等等。什么时候签名者会将讨论中的实体与空间中的位置联系起来？他们把它们放在哪里？如何根据手语的排列调整手语的动作？创建健壮的软件来理解或生成美国手语需要回答诸如此类的问题。PI在这项研究中的目标是发现生成ASL动画的技术，这些动画可以自动预测何时将对话主题与3D位置相关联，将它们放置在哪里，以及这些位置如何影响ASL手势的运动。为此，他将创建来自本地签用者（穿着动作捕捉服和手套）的第一个带注释的ASL运动数据语料库，用与在空间中建立实体表示位置相关的特征注释该语料库，使用机器学习方法分析这些位置建立的时间/地点以及如何在这些位置上参数化符号的3D运动路径，将模型纳入ASL生成软件。招募母语为美国手语的人来评估3D动画效果。这项工作将提高我们对很少被理解但经常出现的美国手语现象的语言知识，因此将为软件产生超出当前技术能力的各种美国手语符号和结构奠定基础。这将导致ASL生成系统产生更高质量的动画，这些动画更符合语法，更容易被聋人用户理解，这将极大地有利于聋人用户和ASL机器翻译的可访问性应用程序。更广泛的影响：ASL动作捕捉语料库和自动处理空间现象的ASL生成器将使更多的计算语言学研究人员能够研究ASL。这项研究也适用于其他国家使用的手语（大多数有类似的现象），以及人类手势动画的生成（在这项工作中开发的经验技术将适用）。PI致力于寻找鼓励聋人高中生从事科学事业的方法，并为聋人学生创造博士研究机会。他将在当地聋人高中用美国手语做关于计算研究的演讲，为聋人高中学生提供暑期研究经验（使用美国手语技能注释语料库，进行评估研究，并向聋人社区介绍计算），招募母语为美国手语手语的人担任博士和本科生研究员，并开设以人为本的计算机科学研究和职业课程（以吸引不同的学生进入该领域），以及辅助技术研究课程（以吸引和培养博士生）。这些教育活动将通过PI的会话美国手语技能，研究与聋哑学生的相关性，以及皇后学院与当地五所聋哑学生高中的独特接近而得以实现。

American Sign Language (ASL) is the primary means of communication for about 500,000 people in the United States. ASL is a distinct language from English; in fact, a majority of deaf U.S. high school graduates have only a fourth-grade (age 10) English reading level. Consequently, many deaf people find it difficult to read English text on computers, in TV captioning, or in other settings. Software to translate English text into an animation of a human character performing ASL would make more information and services accessible to deaf Americans. Unfortunately, however, essential aspects of ASL are not yet modeled by modern computational linguistic software. Specifically, ASL signers associate entities under discussion with 3D locations around their bodies, and the movement of many types of ASL signs changes based on these locations: pronouns, determiners, many noun phrases, many types of verbs, and others. When do signers associate entities under discussion with locations in space? Where do they position them? How must ASL sign movements be modified based on their arrangement? Creation of robust software to understand or generate ASL requires answers to questions such as these. The PI's goal in this research is to discover techniques for generation of ASL animations that automatically predict when to associate conversation topics with 3D locations, where to place them, and how these locations affect ASL sign movements. To these ends, he will create the first annotated corpus of ASL movement data from native signers (in a motion-capture suit and gloves), annotate this corpus with features relating to the establishment of entity-representing locations in space, use machine learning approaches to analyze when/where these locations are established and how 3D motion paths of signs are parameterized on those locations, incorporate the models into ASL generation software, and recruit native ASL signers to evaluate the 3D animations that result. This work will advance our linguistic knowledge relating to little-understood yet frequent ASL phenomena, and so will lay the foundation for software to produce a huge variety of ASL signs and constructions that are beyond the ability of current techniques to generate. This will in turn lead to ASL generation systems that produce higher quality animations that are more grammatical and understandable to deaf users, which will greatly benefit accessibility applications for deaf users and ASL machine translation. Broader Impact: The ASL motion-capture corpus and an ASL generator that automatically handles spatial phenomena will enable more computational linguistic researchers to study ASL. This research also has applications for sign languages used in other countries (most with similar phenomena), and for the generation of animations of human gestures (for which empirical techniques developed in this work will apply). The PI is committed to finding ways to encourage deaf high school students to pursue science careers, and to creating Ph.D. research opportunities for deaf students. He will give presentations in ASL at local deaf high schools about computing research, make available summer research experiences for deaf high school students (using ASL skills to annotate the corpus, conduct evaluation studies, and inform the Deaf community about computing), recruit native ASL signers as Ph.D. and undergraduate researchers, and create courses on people-focused computer science research and careers (to attract diverse students to the field) and on assistive technology research (to interest and train Ph.D. students). These educational activities will be enabled by the PI's conversational ASL skills, by the research's relevance to deaf students, and by the Queens College's unique proximity to five local high schools for deaf students.