EAGER: Collaborative Research: Data Visualizations for Linguistically Annotated, Publicly Shared, Video Corpora for American Sign Language (ASL)

由于缺乏精确的工具来衡量手语中携带关键语法信息的非手动发音(即面部表情和头部手势)，对美国手语的语言学研究一直受到阻碍。到目前为止，同样的限制也阻碍了计算机科学对手语识别和生成的研究。通过之前的NSF支持，PIS创建了宝贵的资源来服务于研究、教育和手语社区，包括：用于分析美国手语(ASL)视频的计算技术和用于手语数据的语言注释的SignStream软件；使用来自本地签名者的视频的大型语言注释和计算分析语料库；以及在线数据访问接口(DAI)，它允许直观和灵活的搜索、浏览和下载，以提供对这些公共共享语料库的轻松访问。他们还利用这些语料库来研究美国手语的语言结构，以及从视频中进行基于计算机的手语识别。最近，他们开发了新版本的SignStream和DAI，带有许多新功能，现在准备公开发布。这两个应用程序都是对这些应用程序早期版本的重大改进，并与大量新的、注释丰富且易于搜索的大型数据集的公开发布相结合，构成了对语言学和计算机科学的研究人员、教育工作者和学生具有重要价值的资源，通过开辟全新的研究途径并使基于计算机的手语识别和生成得到显著改进。由此产生的广泛的研究进展也将有助于未来基于计算机的应用程序，这些应用程序将加强聋人和聋人之间的交流，以及将产生教育效益并全面改善聋人和听力障碍者生活的应用程序。这两个主要软件开发商的兼职工作也将使他们能够提供有限的技术支持，这在SignStream 3和DAI 2公开发布的第一年是必不可少的。该项目的目标是通过结合几个强大的增强和附加功能来进一步改进现有的应用程序，以使共享的工具和数据能够支持语言学(用于分析ASL和其他手语的语言特性)和计算机科学(用于手语识别和生成工作)的新型研究。具体地说，PI将在注释软件和网络界面中将计算机生成的ASL视频分析的图形表示纳入其显示器中，以便用户能够可视化面部表情和头部动作的关键方面的分布和特征，这些关键方面携带手语中的关键语言信息(例如，头部点头和摇动、眉毛高度和眼睛孔径)。手语生成最具挑战性的方面是生成看起来自然、时机恰当的面部表情和头部动作。Metaxas等人最近开发的跟踪和3D建模此类表情的复杂方法。使得从大量视频文件中获取有关这些面部表情和头部手势的准确信息成为可能。

Linguistic research on ASL has been held back by the lack of precise tools for measurement, over large corpora, of the non-manual articulations (i.e., facial expressions and head gestures) that carry key grammatical information in sign languages. The same limitations have, until now, also held back computer science research on sign language recognition and generation. The PIs have created valuable resources, through prior NSF support, to serve the research, education, and sign language communities, including: computational techniques for analysis of American Sign Language (ASL) videos and the SignStream software for linguistic annotation of sign language data; large linguistically annotated and computationally analyzed corpora with videos from native signers; and an online Data Access Interface (DAI) that enables intuitive and flexible searching, browsing, and download, to provide easy access to these publicly shared corpora. They have also exploited these corpora for research on the linguistic structure of ASL and on computer-based sign language recognition from video. Recently, they have developed new versions of SignStream and the DAI with many new features that are now ready to be released publicly. Both represent major improvements over earlier versions of these applications and, combined with the public release of large new richly annotated and readily searchable data sets, constitute resources that will be of great value to researchers, educators, and students in linguistics and computer science, by opening up whole new avenues of research and enabling dramatic improvements in computer-based sign language recognition and generation. The resulting wide-ranging research advances will also contribute to future computer-based applications that will enhance communication for and with deaf individuals, as well as applications that will have educational benefits and overall improve the lives of those who are deaf and hard-of-hearing. The part-time effort to be funded for the two key software developers will also enable them to provide the limited technical support that is essential during the first year of the public release of SignStream 3 and DAI 2.The goal of this project is to further improve the existing applications by incorporating several powerful enhancements and additional functionalities to enable the shared tools and data to support new kinds of research in both linguistics (for analysis of linguistic properties of ASL and other signed languages) and computer science (for work in sign language recognition and generation). Specifically, the PIs will incorporate into their displays, within both the annotation software and the Web interface, graphical representations of computer-generated analyses of ASL videos, so that users will be able to visualize the distribution and characteristics of key aspects of facial expressions and head movements that carry critical linguistic information in sign languages (e.g., head nods and shakes, eyebrow height, and eye aperture). The most challenging aspect of sign language generation has been the production of natural-looking, appropriately timed, facial expressions and head movements. The sophisticated approach to tracking and 3D modeling of such expressions that has been developed recently by Metaxas et al. makes it possible to derive precise information about these facial expressions and head gestures for large sets of video files.