EAGER: Collaborative Research: Data Visualizations for Linguistically Annotated, Publicly Shared, Video Corpora for American Sign Language (ASL)

EAGER：协作研究：美国手语 (ASL) 语言注释、公开共享视频语料库的数据可视化

• 批准号: 1748022
• 负责人: Dimitris Metaxas
• 金额: $ 5.5万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-08-01 至 2018-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1748022&HistoricalAwards=false
• 关键词: EAGER; Collaborative; Research; Data; Visualizations

Linguistic research on ASL has been held back by the lack of precise tools for measurement, over large corpora, of the non-manual articulations (i.e., facial expressions and head gestures) that carry key grammatical information in sign languages. The same limitations have, until now, also held back computer science research on sign language recognition and generation. The PIs have created valuable resources, through prior NSF support, to serve the research, education, and sign language communities, including: computational techniques for analysis of American Sign Language (ASL) videos and the SignStream software for linguistic annotation of sign language data; large linguistically annotated and computationally analyzed corpora with videos from native signers; and an online Data Access Interface (DAI) that enables intuitive and flexible searching, browsing, and download, to provide easy access to these publicly shared corpora. They have also exploited these corpora for research on the linguistic structure of ASL and on computer-based sign language recognition from video. Recently, they have developed new versions of SignStream and the DAI with many new features that are now ready to be released publicly. Both represent major improvements over earlier versions of these applications and, combined with the public release of large new richly annotated and readily searchable data sets, constitute resources that will be of great value to researchers, educators, and students in linguistics and computer science, by opening up whole new avenues of research and enabling dramatic improvements in computer-based sign language recognition and generation. The resulting wide-ranging research advances will also contribute to future computer-based applications that will enhance communication for and with deaf individuals, as well as applications that will have educational benefits and overall improve the lives of those who are deaf and hard-of-hearing. The part-time effort to be funded for the two key software developers will also enable them to provide the limited technical support that is essential during the first year of the public release of SignStream 3 and DAI 2.The goal of this project is to further improve the existing applications by incorporating several powerful enhancements and additional functionalities to enable the shared tools and data to support new kinds of research in both linguistics (for analysis of linguistic properties of ASL and other signed languages) and computer science (for work in sign language recognition and generation). Specifically, the PIs will incorporate into their displays, within both the annotation software and the Web interface, graphical representations of computer-generated analyses of ASL videos, so that users will be able to visualize the distribution and characteristics of key aspects of facial expressions and head movements that carry critical linguistic information in sign languages (e.g., head nods and shakes, eyebrow height, and eye aperture). The most challenging aspect of sign language generation has been the production of natural-looking, appropriately timed, facial expressions and head movements. The sophisticated approach to tracking and 3D modeling of such expressions that has been developed recently by Metaxas et al. makes it possible to derive precise information about these facial expressions and head gestures for large sets of video files.

对美国手语的语言学研究由于缺乏精确的工具来测量大型语料库中的非手动发音（即，面部表情和头部姿势），这些符号携带着手语中的关键语法信息。 到目前为止，同样的限制也阻碍了计算机科学对手语识别和生成的研究。 通过先前的NSF支持，PI创造了宝贵的资源，为研究，教育和手语社区提供服务，包括：用于分析美国手语（ASL）视频的计算技术和用于手语数据语言注释的SignStream软件;大型语言注释和计算分析语料库，其中包括来自本地签名者的视频;以及一个在线数据访问接口（DAI），可实现直观灵活的搜索、浏览和下载，从而轻松访问这些公开共享的语料库。 他们还利用这些语料库研究美国手语的语言结构和基于计算机的视频手语识别。 最近，他们开发了新版本的SignStream和DAI，其中包含许多新功能，现在已经准备好公开发布。 两者都代表了这些应用程序的早期版本的重大改进，并与大型新的丰富注释和易于搜索的数据集的公开发布相结合，构成了对语言学和计算机科学的研究人员，教育工作者和学生具有巨大价值的资源，开辟了全新的研究途径，并使基于计算机的手语识别和生成得到显着改善。 由此产生的广泛的研究进展也将有助于未来基于计算机的应用，这将加强聋人的沟通和与聋人的沟通，以及具有教育效益和全面改善聋人和听力障碍者生活的应用。 关于-为这两个关键软件开发商提供的时间投入也将使他们能够提供有限的技术支持，这在SignStream 3和DAI 2公开发布的第一年是必不可少的。该项目的目标是通过整合几个强大的增强功能和附加功能来进一步改进现有的应用程序，以使共享的工具和数据能够支持新类型的研究，语言学（用于分析美国手语和其他手语的语言特性）和计算机科学（用于手语识别和生成）。 具体而言，PI将在注释软件和Web界面中将计算机生成的ASL视频分析的图形表示纳入其显示中，以便用户能够可视化面部表情和头部运动的关键方面的分布和特征，这些面部表情和头部运动携带手语中的关键语言信息（例如，头部摆动和摇动、眉毛高度和眼睛孔径）。 手语生成最具挑战性的方面一直是自然的外观，适当的时间，面部表情和头部运动的生产。 Metaxas等人最近开发的跟踪和3D建模这种表情的复杂方法使得可以为大量视频文件导出关于这些面部表情和头部姿势的精确信息。

项目成果

NEW shared & interconnected ASL resources: SignStream® 3 Software; DAI 2 for web access to linguistically annotated video corpora; and a sign bank

• 发表时间: 2018-05
• 作者: C. Neidle;Augustine Opoku;G. Dimitriadis;Dimitris N. Metaxas

Toward Personalized Modeling: Incremental and Ensemble Alignment for Sequential Faces in the Wild

• DOI: 10.1007/s11263-017-0996-8
• 发表时间: 2017-02
• 期刊: International Journal of Computer Vision
• 影响因子: 19.5
• 作者: Xi Peng;Shaoting Zhang;Yang Yu;Dimitris N. Metaxas

Linguistically-driven Framework for Computationally Efficient and Scalable Sign Recognition

• 发表时间: 2018-05
• 作者: Dimitris N. Metaxas;Mark Dilsizian;C. Neidle

Scalable ASL Sign Recognition using Model-based Machine Learning and Linguistically Annotated Corpora

• 期刊: IEEE Access
• 影响因子: 3.9
• 作者: Dimitris N. Metaxas;Mark Dilsizian;C. Neidle