EAGER: Converting Print Dictionaries to Machine-Interpretable Format

字典记录了一门语言的组成部分——它的单词和习惯短语，以及它们的发音、语法特性、意义和用法的描述，是语言文献的一个重要组成部分，还有参考语法、转录文本和录音。直到最近，词典都是手工编纂和组织的，输入某种排版系统，最后以印刷形式呈现给学者和语言学习者使用。现代词典现在都是电子编纂和组织的，因此它们所包含的信息不仅可以用来制作独立的印刷品，而且还可以与其他组件集成在一起，以确保文档作为一个整体的更高准确性，使更新能够定期产生，并支持为以这种方式记录的语言开发自然语言处理工具。这个探索性项目的目标是开发一种方法，使机器能够理解现存的数百种濒危语言和其他低资源语言的印刷词典的隐含结构，这是使它们的文档对后代最有用的关键的第一步。打印字典使用排序、字体和其他格式约定来指示字典条目的预期结构。该项目的首要任务是使用光学字符读取（OCR）软件将这些条目转换为机器可解释的形式，以保留原始格式。第二步是开发软件，将校正后的OCR输出转换成结构化的、机器可解释的存档标准格式。由于不同字典的打印格式差异很大，因此需要人工干预来告知软件如何将特定字典条目的隐式表示转换为显式表示。但是，这种手动注释只需要字典的一小部分，因为所有条目的格式约定都是一致的，一旦学会了，就可以用来识别和纠正错误和不一致，并启用自动编辑任务，如更新正字法。该工具将使用二十世纪后半叶出版的拉丁美洲两种土著语言的印刷版词典进行开发、测试和评价。该项目由行为与认知科学部的濒危语言记录项目和信息与智能系统部的稳健智能项目共同支持。

A dictionary documents the building blocks of a language -- its words and idiomatic phrases, with descriptions of their pronunciations, grammatical properties, meanings and uses -- and is an essential component of language documentation, together with a reference grammar and transcribed texts and recordings. Until recently, dictionaries were compiled and organized by hand, entered into some kind of typesetting system, and finally rendered in print form for use by scholars and language learners. Contemporary dictionaries are now compiled and organized electronically so that the information they contain can be used not only to produce stand-alone print artifacts, but also be integrated with the other components to ensure greater accuracy of the documentation as a whole, enable updates to be produced at regular intervals, and support the development of natural-language processing tools for the languages that are documented in this way. The goal of this exploratory project is to develop methods for machines to understand the implicit structure of the hundreds of extant print dictionaries of endangered and other low-resource languages as a critical first step in enabling their documentation to be of maximal usefulness to future generations.Print dictionaries use ordering, typeface and other formatting conventions to indicate the intended structure of dictionary entries. The first task of this project is to use optical character reading (OCR) software to convert those entries to machine-interpretable form so as to preserve the original formatting. The second is to develop software to convert the corrected OCR output into structured, machine-interpretable archive-standard formats. Because print formats vary widely across dictionaries, human intervention is required to inform the software about how to translate the implicit representations for a particular dictionary's entries into explicit ones. But such manual annotation is only required for a small part of the dictionary, as the formatting conventions are consistent across all of its entries, and once learned can be used to identify and correct errors and inconsistencies, and enable automated editing tasks like updating orthographies. The tool will be developed, tested and evaluated using print dictionaries of two indigenous languages of Latin America that were produced in the latter part of the twentieth century. This project is jointly supported by the Documenting Endangered Languages Program in the Behavioral and Cognitive Sciences Division and by the Robust Intelligence Program in the Information and Intelligent Systems Division.