EAGER: Collaborative Research: Data Visualizations for Linguistically Annotated, Publicly Shared, Video Corpora for American Sign Language (ASL)

EAGER：协作研究：美国手语 (ASL) 语言注释、公开共享视频语料库的数据可视化

• 批准号: 1748022
• 负责人: Dimitris Metaxas
• 金额: $ 5.5万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-08-01 至 2018-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1748022&HistoricalAwards=false
• 关键词: EAGER; Collaborative; Research; Data; Visualizations

Linguistic research on ASL has been held back by the lack of precise tools for measurement, over large corpora, of the non-manual articulations (i.e., facial expressions and head gestures) that carry key grammatical information in sign languages. The same limitations have, until now, also held back computer science research on sign language recognition and generation. The PIs have created valuable resources, through prior NSF support, to serve the research, education, and sign language communities, including: computational techniques for analysis of American Sign Language (ASL) videos and the SignStream software for linguistic annotation of sign language data; large linguistically annotated and computationally analyzed corpora with videos from native signers; and an online Data Access Interface (DAI) that enables intuitive and flexible searching, browsing, and download, to provide easy access to these publicly shared corpora. They have also exploited these corpora for research on the linguistic structure of ASL and on computer-based sign language recognition from video. Recently, they have developed new versions of SignStream and the DAI with many new features that are now ready to be released publicly. Both represent major improvements over earlier versions of these applications and, combined with the public release of large new richly annotated and readily searchable data sets, constitute resources that will be of great value to researchers, educators, and students in linguistics and computer science, by opening up whole new avenues of research and enabling dramatic improvements in computer-based sign language recognition and generation. The resulting wide-ranging research advances will also contribute to future computer-based applications that will enhance communication for and with deaf individuals, as well as applications that will have educational benefits and overall improve the lives of those who are deaf and hard-of-hearing. The part-time effort to be funded for the two key software developers will also enable them to provide the limited technical support that is essential during the first year of the public release of SignStream 3 and DAI 2.The goal of this project is to further improve the existing applications by incorporating several powerful enhancements and additional functionalities to enable the shared tools and data to support new kinds of research in both linguistics (for analysis of linguistic properties of ASL and other signed languages) and computer science (for work in sign language recognition and generation). Specifically, the PIs will incorporate into their displays, within both the annotation software and the Web interface, graphical representations of computer-generated analyses of ASL videos, so that users will be able to visualize the distribution and characteristics of key aspects of facial expressions and head movements that carry critical linguistic information in sign languages (e.g., head nods and shakes, eyebrow height, and eye aperture). The most challenging aspect of sign language generation has been the production of natural-looking, appropriately timed, facial expressions and head movements. The sophisticated approach to tracking and 3D modeling of such expressions that has been developed recently by Metaxas et al. makes it possible to derive precise information about these facial expressions and head gestures for large sets of video files.

由于缺乏精确的工具来测量大型语料库中携带手语关键语法信息的非手动发音（即面部表情和头部手势），美国手语的语言学研究一直受到阻碍。到目前为止，同样的限制也阻碍了计算机科学对手语识别和生成的研究。通过NSF的支持，pi已经创造了宝贵的资源，为研究、教育和手语社区服务，包括：用于分析美国手语（ASL）视频的计算技术和用于手语数据语言注释的SignStream软件；大型语言注释和计算分析语料库与视频从本地签名者；和一个在线数据访问接口（DAI），使直观和灵活的搜索，浏览和下载，提供方便的访问这些公开共享的语料库。他们还利用这些语料库研究了美国手语的语言结构和基于计算机的视频手语识别。最近，他们开发了带有许多新特性的SignStream和DAI的新版本，现在准备向公众发布。两者都代表了对这些应用程序早期版本的重大改进，并且结合公开发布的大量新的丰富注释和易于搜索的数据集，构成了对语言学和计算机科学的研究人员，教育工作者和学生具有巨大价值的资源，开辟了全新的研究途径，并使基于计算机的手语识别和生成得到了巨大的改进。由此产生的广泛的研究进展也将有助于未来以计算机为基础的应用，这些应用将加强聋哑人的交流和与聋哑人的交流，以及具有教育效益的应用，并全面改善聋哑人和听力障碍者的生活。为两位主要软件开发人员提供的兼职工作也将使他们能够在公开发布SignStream 3和DAI 2的第一年提供必要的有限技术支持。该项目的目标是通过整合一些强大的增强和额外的功能来进一步改进现有的应用程序，使共享的工具和数据能够支持语言学（用于分析美国手语和其他手语的语言特性）和计算机科学（用于手语识别和生成）的新型研究。具体来说，pi将在注释软件和网络界面中，将计算机生成的美国手语视频分析的图形表示合并到他们的显示中，以便用户能够可视化面部表情和头部动作的关键方面的分布和特征，这些面部表情和头部动作携带着手语的关键语言信息（例如，点头和摇头，眉毛高度和眼睛的宽度）。手语生成最具挑战性的方面是产生自然的，适当的时间，面部表情和头部动作。Metaxas等人最近开发了一种复杂的面部表情跟踪和3D建模方法，可以从大量视频文件中获得关于这些面部表情和头部手势的精确信息。

项目成果

NEW shared & interconnected ASL resources: SignStream® 3 Software; DAI 2 for web access to linguistically annotated video corpora; and a sign bank

• 发表时间: 2018-05
• 作者: C. Neidle;Augustine Opoku;G. Dimitriadis;Dimitris N. Metaxas

Toward Personalized Modeling: Incremental and Ensemble Alignment for Sequential Faces in the Wild

• DOI: 10.1007/s11263-017-0996-8
• 发表时间: 2017-02
• 期刊: International Journal of Computer Vision
• 影响因子: 19.5
• 作者: Xi Peng;Shaoting Zhang;Yang Yu;Dimitris N. Metaxas

Linguistically-driven Framework for Computationally Efficient and Scalable Sign Recognition

• 发表时间: 2018-05
• 作者: Dimitris N. Metaxas;Mark Dilsizian;C. Neidle

Scalable ASL Sign Recognition using Model-based Machine Learning and Linguistically Annotated Corpora

• 期刊: IEEE Access
• 影响因子: 3.9
• 作者: Dimitris N. Metaxas;Mark Dilsizian;C. Neidle