EAGER: Collaborative Research: Data Visualizations for Linguistically Annotated, Publicly Shared, Video Corpora for American Sign Language (ASL)

EAGER：协作研究：美国手语 (ASL) 语言注释、公开共享视频语料库的数据可视化

• 批准号: 1748022
• 负责人: Dimitris Metaxas
• 金额: $ 5.5万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-08-01 至 2018-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1748022&HistoricalAwards=false
• 关键词: EAGER; Collaborative; Research; Data; Visualizations

Linguistic research on ASL has been held back by the lack of precise tools for measurement, over large corpora, of the non-manual articulations (i.e., facial expressions and head gestures) that carry key grammatical information in sign languages. The same limitations have, until now, also held back computer science research on sign language recognition and generation. The PIs have created valuable resources, through prior NSF support, to serve the research, education, and sign language communities, including: computational techniques for analysis of American Sign Language (ASL) videos and the SignStream software for linguistic annotation of sign language data; large linguistically annotated and computationally analyzed corpora with videos from native signers; and an online Data Access Interface (DAI) that enables intuitive and flexible searching, browsing, and download, to provide easy access to these publicly shared corpora. They have also exploited these corpora for research on the linguistic structure of ASL and on computer-based sign language recognition from video. Recently, they have developed new versions of SignStream and the DAI with many new features that are now ready to be released publicly. Both represent major improvements over earlier versions of these applications and, combined with the public release of large new richly annotated and readily searchable data sets, constitute resources that will be of great value to researchers, educators, and students in linguistics and computer science, by opening up whole new avenues of research and enabling dramatic improvements in computer-based sign language recognition and generation. The resulting wide-ranging research advances will also contribute to future computer-based applications that will enhance communication for and with deaf individuals, as well as applications that will have educational benefits and overall improve the lives of those who are deaf and hard-of-hearing. The part-time effort to be funded for the two key software developers will also enable them to provide the limited technical support that is essential during the first year of the public release of SignStream 3 and DAI 2.The goal of this project is to further improve the existing applications by incorporating several powerful enhancements and additional functionalities to enable the shared tools and data to support new kinds of research in both linguistics (for analysis of linguistic properties of ASL and other signed languages) and computer science (for work in sign language recognition and generation). Specifically, the PIs will incorporate into their displays, within both the annotation software and the Web interface, graphical representations of computer-generated analyses of ASL videos, so that users will be able to visualize the distribution and characteristics of key aspects of facial expressions and head movements that carry critical linguistic information in sign languages (e.g., head nods and shakes, eyebrow height, and eye aperture). The most challenging aspect of sign language generation has been the production of natural-looking, appropriately timed, facial expressions and head movements. The sophisticated approach to tracking and 3D modeling of such expressions that has been developed recently by Metaxas et al. makes it possible to derive precise information about these facial expressions and head gestures for large sets of video files.

关于ASL的语言研究，由于缺乏在大型语料库中，缺乏精确的测量工具（即面部表情和头部手势），这些工具具有符号语言的关键语法信息。 到目前为止，同样的限制也阻碍了计算机科学研究对手语识别和发电的研究。 PI通过先前的NSF支持创建了宝贵的资源，以服务于研究，教育和手语社区，包括：用于分析美国手语（ASL）视频的计算技术和用于语言语言数据的语言注释的Signstream软件；大型语言注释和计算分析的语料库，并通过本地签名者的视频进行了视频；以及一个在线数据访问接口（DAI），可实现直观且灵活的搜索，浏览和下载，以便轻松访问这些公开共享的Corpora。 他们还利用了这些语料库来研究ASL的语言结构以及视频中基于计算机的手语识别。 最近，他们开发了Signstream和Dai的新版本，并具有许多新功能，这些功能现在可以公开发布。 两者都代表了对这些应用程序的早期版本的重大改进，再加上公众发布大型新的注释和易于搜索的数据集，构成了对研究人员，教育工作者和计算机科学领域的重要价值，通过开放全新的研究途径，并在基于基于计算机的标志语言知名度和生成中实现巨大的改进。 由此产生的广泛研究进步还将为未来的基于计算机的应用程序做出贡献，这些应用程序将增强聋人和聋人的沟通，以及将具有教育益处并总体上改善聋哑人和听力障碍者的生活的应用程序。 为两个关键软件开发人员提供资金的兼职努力还将使他们能够提供有限的技术支持，在公开发布Signstream 3和Dai 2的第一年，这是必不可少的，该项目的目标是该项目的目的是通过将几种强大的增强功能和其他功能纳入到同种语言的属性中，以实现各种语言的属性（用于分析），以分析语言的新功能，以进一步改善现有的应用程序，以实现各种语言和数据，以实现各种语言和数据。手语识别和发电）。 Specifically, the PIs will incorporate into their displays, within both the annotation software and the Web interface, graphical representations of computer-generated analyses of ASL videos, so that users will be able to visualize the distribution and characteristics of key aspects of facial expressions and head movements that carry critical linguistic information in sign languages (e.g., head nods and shakes, eyebrow height, and eye aperture). 手语产生最具挑战性的方面是生产自然，适当时机，面部表情和头部运动。 Metaxas等人最近开发的此类表达式的跟踪和3D建模的复杂方法。可以为大量视频文件提供有关这些面部表情和头部手势的精确信息。

项目成果

Toward Personalized Modeling: Incremental and Ensemble Alignment for Sequential Faces in the Wild

• DOI: 10.1007/s11263-017-0996-8
• 发表时间: 2017-02
• 期刊: International Journal of Computer Vision
• 影响因子: 19.5
• 作者: Xi Peng;Shaoting Zhang;Yang Yu;Dimitris N. Metaxas