CAREER: Learning to Generate American Sign Language Animation through Motion-Capture and Participation of Native ASL Signers

美国手语（ASL）是美国约50万人的主要交流手段。 美国手语是一种与英语截然不同的语言;事实上，大多数聋人美国高中毕业生只有四年级（10岁）的英语阅读水平。 因此，许多聋人发现很难在电脑上阅读英语文本，在电视字幕，或在其他设置。 将英语文本翻译成动画的软件将使聋人获得更多的信息和服务。 然而，不幸的是，ASL的基本方面还没有被现代计算语言软件建模。 具体来说，ASL签名者将讨论中的实体与其身体周围的3D位置相关联，并且许多类型的ASL签名的移动基于这些位置而改变：代词、限定词、许多名词短语、许多类型的动词等。 签名者何时将讨论中的实体与空间中的位置相关联？ 他们把它们放在哪里？ 美国手语的标志动作必须如何根据它们的排列进行修改？ 创建强大的软件来理解或生成ASL需要回答这些问题。 PI在这项研究中的目标是发现生成ASL动画的技术，这些动画可以自动预测何时将对话主题与3D位置相关联，将它们放置在哪里，以及这些位置如何影响ASL标志运动。 为此，他将从本地签名者那里创建第一个带注释的ASL运动数据语料库（在运动捕捉套装和手套中），用与空间中的实体表示位置的建立有关的特征来注释该语料库，使用机器学习方法来分析何时/在何处建立这些位置以及如何在这些位置上参数化标志的3D运动路径，将模型整合到ASL生成软件中，并招募本地ASL签名者来评估生成的3D动画。 这项工作将推进我们的语言知识，涉及到很少理解但频繁的ASL现象，因此将奠定基础的软件，以产生大量的各种各样的ASL符号和结构，超出了目前的技术生成的能力。 这反过来又会导致ASL生成系统产生更高质量的动画，这些动画对聋人用户来说更符合语法和更容易理解，这将极大地有利于聋人用户的无障碍应用程序和ASL机器翻译。 更广泛的影响：手语动作捕捉语料库和自动处理空间现象的手语生成器将使更多的计算语言学研究者能够研究手语。 这项研究也适用于在其他国家使用的手语（大多数具有类似的现象），并用于生成人类手势的动画（在这项工作中开发的经验技术将适用）。 PI致力于寻找鼓励聋哑高中生追求科学事业的方法，并创造博士学位。为聋哑学生提供研究机会。 他将在当地聋人高中就计算研究发表演讲，为聋人高中生提供暑期研究经验（使用ASL技能注释语料库，进行评估研究，并告知聋人社区有关计算），招募本地ASL签名者作为博士。和本科研究人员，并创建以人为本的计算机科学研究和职业课程（以吸引不同的学生到该领域）和辅助技术研究（感兴趣和培养博士学位）。学生）。 这些教育活动将通过PI的美国手语对话技能、研究与聋人学生的相关性以及皇后学院与当地五所聋人学生高中的独特邻近性来实现。

American Sign Language (ASL) is the primary means of communication for about 500,000 people in the United States. ASL is a distinct language from English; in fact, a majority of deaf U.S. high school graduates have only a fourth-grade (age 10) English reading level. Consequently, many deaf people find it difficult to read English text on computers, in TV captioning, or in other settings. Software to translate English text into an animation of a human character performing ASL would make more information and services accessible to deaf Americans. Unfortunately, however, essential aspects of ASL are not yet modeled by modern computational linguistic software. Specifically, ASL signers associate entities under discussion with 3D locations around their bodies, and the movement of many types of ASL signs changes based on these locations: pronouns, determiners, many noun phrases, many types of verbs, and others. When do signers associate entities under discussion with locations in space? Where do they position them? How must ASL sign movements be modified based on their arrangement? Creation of robust software to understand or generate ASL requires answers to questions such as these. The PI's goal in this research is to discover techniques for generation of ASL animations that automatically predict when to associate conversation topics with 3D locations, where to place them, and how these locations affect ASL sign movements. To these ends, he will create the first annotated corpus of ASL movement data from native signers (in a motion-capture suit and gloves), annotate this corpus with features relating to the establishment of entity-representing locations in space, use machine learning approaches to analyze when/where these locations are established and how 3D motion paths of signs are parameterized on those locations, incorporate the models into ASL generation software, and recruit native ASL signers to evaluate the 3D animations that result. This work will advance our linguistic knowledge relating to little-understood yet frequent ASL phenomena, and so will lay the foundation for software to produce a huge variety of ASL signs and constructions that are beyond the ability of current techniques to generate. This will in turn lead to ASL generation systems that produce higher quality animations that are more grammatical and understandable to deaf users, which will greatly benefit accessibility applications for deaf users and ASL machine translation. Broader Impact: The ASL motion-capture corpus and an ASL generator that automatically handles spatial phenomena will enable more computational linguistic researchers to study ASL. This research also has applications for sign languages used in other countries (most with similar phenomena), and for the generation of animations of human gestures (for which empirical techniques developed in this work will apply). The PI is committed to finding ways to encourage deaf high school students to pursue science careers, and to creating Ph.D. research opportunities for deaf students. He will give presentations in ASL at local deaf high schools about computing research, make available summer research experiences for deaf high school students (using ASL skills to annotate the corpus, conduct evaluation studies, and inform the Deaf community about computing), recruit native ASL signers as Ph.D. and undergraduate researchers, and create courses on people-focused computer science research and careers (to attract diverse students to the field) and on assistive technology research (to interest and train Ph.D. students). These educational activities will be enabled by the PI's conversational ASL skills, by the research's relevance to deaf students, and by the Queens College's unique proximity to five local high schools for deaf students.