DC: Large: Collaborative Research: Mining a Million Scanned Books: Linguistic and Structure Analysis, Fast Expanded Search, and Improved OCR

马萨诸塞州阿默斯特大学的智能信息检索中心、塔夫茨的珀尔修斯数字图书馆项目和互联网档案馆正在研究数字化图书收藏的大规模信息提取和检索技术。为了向学者和公众提供有效的分析和搜索，并处理这些集合的多样性和规模，本项目重点改进了七项连锁技术：通过单词识别提高OCR准确性，使用特征的联合分布创建概率模型，并建立跨文档的特定主题语言模型；结构元数据提取，以挖掘标题、章节、目录和索引；语言分析和信息提取，以对嘈杂的OCR输出执行句法分析和实体提取；推断文档关系结构，以挖掘引文、引文、翻译和释义；通过时间进行潜在主题建模，以改进OCR和检索的语言建模，并跟踪跨时期和流派的思想传播；对相关性模型的查询扩展，通过对文件比较的离线预处理来提高信息检索的相关性；以及探索性数据分析接口，向文件集合的用户提供高效工具，以更新重要实体、事件、主题和语言特征的复杂模型。当应用于大型语料库时，这些技术相辅相成：改进的主题建模使OCR的语言模型更有针对性；提取结构元数据改进了引文分析；实体提取改进了主题建模和查询扩展。该项目的试验台是来自互联网档案馆的不断增长的100多万本开放获取图书的语料库。

The Center for Intelligent Information Retrieval at UMass Amherst, the Perseus Digital Library Project at Tufts, and the Internet Archive are investigating large-scale information extraction and retrieval technologies for digitized book collections. To provide effective analysis and search for scholars and the general public, and to handle the diversity and scale of these collections, this project focuses on improvements in seven interlocking technologies: improved OCR accuracy through word spotting, creating probabilistic models using joint distributions of features, and building topic-specific language models across documents; structural metadata extraction, to mine headers, chapters, tables of contents, and indices; linguistic analysis and information extraction, to perform syntactic analysis and entity extraction on noisy OCR output; inferred document relational structure, to mine citations, quotations, translations, and paraphrases; latent topic modeling through time, to improve language modeling for OCR and retrieval, and to track the spread of ideas across periods and genres; query expansion for relevance models, to improve relevance in information retrieval by offline pre-processing of document comparisons; and interfaces for exploratory data analysis, to provide users of the document collection with efficient tools to update complex models of important entities, events, topics, and linguistic features. When applied across large corpora, these technologies reinforce each other: improved topic modeling enables more targeted language models for OCR; extracting structural metadata improves citation analysis; and entity extraction improves topic modeling and query expansion.The testbed for this project is the growing corpus of over one million open-access books from the Internet Archive.