Collaborative Research: CI-ADDO-EN: Development of Publicly Available, Easily Searchable, Linguistically Analyzed, Video Corpora for Sign Language and Gesture Research

点击翻译按钮获取中文摘要

The goal of this project is to create a linguistically annotated, publicly available, and easily searchable corpus of video from American Sign Language (ASL). This will constitute an important piece of infrastructure, enabling new kinds of research in both linguistics and vision-based recognition of ASL. In addition, a key goal is to make this corpus easily accessible to the broader ASL community, including users and learners of ASL. As a result of our long-term efforts, we have an extensive collection of linguistically annotated video data from native signers of ASL. However, the potential value of these corpora has been largely untapped, notwithstanding their extensive and productive use by our team and others. Existing limitations in our hardware and software infrastructure make it cumbersome to search and identify data of interest, and to share data among our institutions and with other researchers. In this project, we propose hardware and software innovations that will constitute a major qualitative upgrade in the organization, searchability, and public availability of the existing (and expanding) corpus. The enhancement and improved Web-accessibility of these corpora will be invaluable for linguistic research, enabling new kinds of discoveries and the testing of hypotheses that would otherwise have be difficult to investigate. On the computer vision side, the proposed new annotations will provide an extensive public dataset for training and benchmarking a variety of computer vision algorithms. This will facilitate research and expedite progress in gesture recognition, hand pose estimation, human tracking, and large vocabulary, and continuous ASL recognition. Furthermore, this dataset will be useful as training and benchmarking data for algorithms in the broader areas of computer vision, machine learning, and similarity-based indexing. The advances in linguistic knowledge about ASL and in computer-based ASL recognition that will be accelerated by the availability of resources of the kind proposed here will contribute to development of technologies for education and universal access. For example, tools for searching collections of ASL video for occurrences of specific signs, or converting ASL signing to English, are still far from attaining the level of functionality and usability to which users are accustomed for spoken/written languages. Our corpora will enable research that aims to bring such vision-based ASL recognition applications closer to reality. Moreover, these resources will afford important opportunities to individuals who would not otherwise be in a position to conduct such research (e.g., for lack of access to native ASL signers or high-quality synchronized video equipment, or lack of resources/expertise to carry out extensive linguistic annotations). Making our corpora available online will also allow the broader community of ASL users to access our data directly. Students of ASL will be able to retrieve video showing examples of a specific sign used in actual sentences, or examples of a grammatical construction. ASL instructors and teachers of the Deaf will also have easy access to video examples of lexical items and grammatical constructions as used by a variety of native signers for use in language instruction and evaluation. Thus, the proposed web interface to our data collection will be a useful educational resource for users, teachers, and learners of ASL.