Collaborative Research: CI-ADDO-EN: Development of Publicly Available, Easily Searchable, Linguistically Analyzed, Video Corpora for Sign Language and Gesture Research

这个项目的目标是创建一个带有语言注释的、公开的、易于搜索的美国手语(ASL)视频语料库。这将构成一个重要的基础设施，使语言学和基于视觉的ASL识别的新型研究成为可能。此外，一个关键目标是使更广泛的美国手语社区，包括美国手语的用户和学习者，能够轻松地访问该语料库。作为我们长期努力的结果，我们从ASL的本地签名者那里收集了大量带有语言注释的视频数据。然而，尽管我们的团队和其他人广泛而富有成效地使用了这些语料库，但它们的潜在价值在很大程度上尚未开发。我们硬件和软件基础设施的现有限制使搜索和识别感兴趣的数据以及在我们的机构之间和与其他研究人员共享数据变得困难。在这个项目中，我们提出了硬件和软件创新，这将构成现有(和不断扩大的)语料库的组织、可搜索性和公共可用性方面的重大质量升级。这些语料库的增强和网络可访问性的提高对语言学研究将是无价的，这将使新的发现和假说的检验成为可能，否则这些假说将很难研究。在计算机视觉方面，拟议的新注释将提供一个广泛的公共数据集，用于培训和基准各种计算机视觉算法。这将促进手势识别、手势估计、人体跟踪和大词汇量以及连续ASL识别方面的研究和加快进展。此外，该数据集将作为计算机视觉、机器学习和基于相似性的索引等更广泛领域中的算法的训练和基准数据。关于ASL的语言知识和以计算机为基础的ASL识别方面的进步，将因这里提议的这类资源的可获得性而加速，这将有助于教育和普及技术的发展。例如，用于搜索ASL视频集合以查找特定手势出现的工具，或将ASL手语转换为英语的工具，仍远未达到用户习惯使用口语/书面语的功能和可用性水平。我们的语料库将使旨在使这种基于视觉的ASL识别应用更接近现实的研究成为可能。此外，这些资源将为本来无法进行这类研究的个人提供重要机会(例如，无法接触到当地的美国手语签名者或高质量的同步视频设备，或缺乏资源/专业知识来进行广泛的语言注释)。使我们的语料库在线可用也将允许更广泛的ASL用户社区直接访问我们的数据。ASL的学生将能够检索视频，展示实际句子中使用的特定符号的例子，或语法结构的例子。美国手语教师和聋人教师还可以很容易地获得各种母语手语者在语言教学和评估中使用的词汇和语法结构的视频范例。因此，对于美国手语的用户、教师和学习者来说，拟议的数据收集网络界面将是一个有用的教育资源。

The goal of this project is to create a linguistically annotated, publicly available, and easily searchable corpus of video from American Sign Language (ASL). This will constitute an important piece of infrastructure, enabling new kinds of research in both linguistics and vision-based recognition of ASL. In addition, a key goal is to make this corpus easily accessible to the broader ASL community, including users and learners of ASL. As a result of our long-term efforts, we have an extensive collection of linguistically annotated video data from native signers of ASL. However, the potential value of these corpora has been largely untapped, notwithstanding their extensive and productive use by our team and others. Existing limitations in our hardware and software infrastructure make it cumbersome to search and identify data of interest, and to share data among our institutions and with other researchers. In this project, we propose hardware and software innovations that will constitute a major qualitative upgrade in the organization, searchability, and public availability of the existing (and expanding) corpus. The enhancement and improved Web-accessibility of these corpora will be invaluable for linguistic research, enabling new kinds of discoveries and the testing of hypotheses that would otherwise have be difficult to investigate. On the computer vision side, the proposed new annotations will provide an extensive public dataset for training and benchmarking a variety of computer vision algorithms. This will facilitate research and expedite progress in gesture recognition, hand pose estimation, human tracking, and large vocabulary, and continuous ASL recognition. Furthermore, this dataset will be useful as training and benchmarking data for algorithms in the broader areas of computer vision, machine learning, and similarity-based indexing. The advances in linguistic knowledge about ASL and in computer-based ASL recognition that will be accelerated by the availability of resources of the kind proposed here will contribute to development of technologies for education and universal access. For example, tools for searching collections of ASL video for occurrences of specific signs, or converting ASL signing to English, are still far from attaining the level of functionality and usability to which users are accustomed for spoken/written languages. Our corpora will enable research that aims to bring such vision-based ASL recognition applications closer to reality. Moreover, these resources will afford important opportunities to individuals who would not otherwise be in a position to conduct such research (e.g., for lack of access to native ASL signers or high-quality synchronized video equipment, or lack of resources/expertise to carry out extensive linguistic annotations). Making our corpora available online will also allow the broader community of ASL users to access our data directly. Students of ASL will be able to retrieve video showing examples of a specific sign used in actual sentences, or examples of a grammatical construction. ASL instructors and teachers of the Deaf will also have easy access to video examples of lexical items and grammatical constructions as used by a variety of native signers for use in language instruction and evaluation. Thus, the proposed web interface to our data collection will be a useful educational resource for users, teachers, and learners of ASL.