DC: Large: Collaborative Research: Mining a Million Scanned Books: Linguistic and Structure Analysis, Fast Expanded Search, and Improved OCR

马萨诸塞大学阿默斯特分校的智能信息检索中心、塔夫茨大学的珀尔修斯数字图书馆项目和互联网档案馆正在研究数字化藏书的大规模信息提取和检索技术。为了给学者和公众提供有效的分析和搜索，并处理这些集合的多样性和规模，该项目着重于改进七种互锁技术：通过单词识别提高OCR的准确性，使用特征的联合分布创建概率模型，以及跨文档构建特定主题的语言模型；结构化元数据提取，用于挖掘标题、章节、目录和索引；语言分析和信息提取，对有噪声的OCR输出进行句法分析和实体提取；推断文档关系结构，以挖掘引用、引用、翻译和释义；通过时间进行潜在主题建模，改进OCR和检索的语言建模，并跟踪思想在不同时期和类型的传播；对相关模型进行查询扩展，通过对文档比较进行离线预处理，提高信息检索中的相关性；以及用于探索性数据分析的接口，为文档集合的用户提供有效的工具来更新重要实体、事件、主题和语言特征的复杂模型。当应用于大型语料库时，这些技术相互加强：改进的主题建模为OCR提供了更有针对性的语言模型；提取结构元数据提高引文分析；实体提取改进了主题建模和查询扩展。这个项目的测试平台是来自互联网档案馆的100多万本开放获取的图书。

The Center for Intelligent Information Retrieval at UMass Amherst, the Perseus Digital Library Project at Tufts, and the Internet Archive are investigating large-scale information extraction and retrieval technologies for digitized book collections. To provide effective analysis and search for scholars and the general public, and to handle the diversity and scale of these collections, this project focuses on improvements in seven interlocking technologies: improved OCR accuracy through word spotting, creating probabilistic models using joint distributions of features, and building topic-specific language models across documents; structural metadata extraction, to mine headers, chapters, tables of contents, and indices; linguistic analysis and information extraction, to perform syntactic analysis and entity extraction on noisy OCR output; inferred document relational structure, to mine citations, quotations, translations, and paraphrases; latent topic modeling through time, to improve language modeling for OCR and retrieval, and to track the spread of ideas across periods and genres; query expansion for relevance models, to improve relevance in information retrieval by offline pre-processing of document comparisons; and interfaces for exploratory data analysis, to provide users of the document collection with efficient tools to update complex models of important entities, events, topics, and linguistic features. When applied across large corpora, these technologies reinforce each other: improved topic modeling enables more targeted language models for OCR; extracting structural metadata improves citation analysis; and entity extraction improves topic modeling and query expansion.The testbed for this project is the growing corpus of over one million open-access books from the Internet Archive.