Essential Tools for Computational Research on Visual-Gestural Language

这是一个为期三年的持续奖励的第一年资助。手语的识别和生成以及口语的手势成分的研究一直受到阻碍，因为没有大规模的语言注释语料库，而这种语料库导致了口语领域的重大进展。制作这种语料库的一个主要障碍是缺乏计算机工具来协助有效地分析和转录视觉语言数据。在这个项目中，PI和她的团队将开发必要的工具和技术，以支持精细详细的语音信息的有效转录，并将其与计算算法提供的信息集成，从而能够创建大规模的语料库，在计算机科学研究所需的粒度水平上进行注释。还将开发用于转录过程的几个方面的半自动化的基于机器视觉的算法。这项研究将导致语料库的生产和公共传播，其中将包括对美国手语视频数据的细粒度语言注释。这些语料库的可用性和用于分析视觉数据的计算工具的改进对于手语和口语的手势成分的语言学研究将是无价的。该语料库将用于训练计算机模型，从而在手语和手势的识别和生成方面取得进展。这些工具将用于美国手语和其他手语的教学。最终，这项研究将对人机界面产生影响，使用户能够通过手语或通过语音和手势的组合与计算机进行交互，并可能为残疾用户提供替代输入机制，以及自动识别和分类人类行为的技术。

This is the first-year funding of a three year continuing award. Research on recognition and generation of signed languages and the gestural component of spoken languages has been held back by the unavailability of large-scale linguistically annotated corpora of the kind that led to significant advances in the area of spoken language. A major obstacle to the production of such corpora has been the lack of computational tools to assist in efficient analysis and transcription of visual language data. In this project the PI and her team will develop the tools and techniques necessary to support efficient transcription of finely detailed phonological information and its integration with information provided by computational algorithms, so as to enable the creation of large-scale corpora annotated at the level of granularity essential for computer science research. Machine vision-based algorithms for semi-automation of several aspects of the transcription process will also be developed.This research will result in the production and public dissemination of corpora that will include fine-grained linguistic annotations of ASL video data. The availability of these corpora and the refinement of computational tools for analyzing visual data will be invaluable for the linguistic study of signed languages and the gestural component of spoken languages. The corpora will be used in training computer models, thereby leading to advances in both recognition and generation of signed languages and gesture. The tools will have educational applications for the teaching of ASL and other signed languages. Ultimately, this research will have implications for human-computer interfaces, enabling users to interact with computers via sign language or via a combination of speech and gesture, and may also lead to alternate input mechanisms for disabled users as well as techniques for automatic recognition and classification of human actions.