A Document Processing System

文档处理系统

• 批准号: 9160906
• 负责人: Willy Wilbur
• 金额: $ 43.97万
• 依托单位: NATIONAL LIBRARY OF MEDICINE
• 依托单位国家: 美国
• 资助国家: 美国
• 起止时间: 至
• 项目状态: 未结题
• 来源: https://reporter.nih.gov/project-details/9160906
• 关键词: Code; Data; Databases; Environment; Frequencies; Literature; MEDLINE; Machine Learning; Maps; MeSH Thesaurus; Memory; Methods; Process; Programming Languages; PubMed; Records; Research; Retrieval; Speed; Structure; System; Testing; Text; Update; Work; abstracting; base; computerized data processing; improved; phrases; repository; software development; software systems; vector

A system of C++ language programs has been developed for the purpose of finding the closely related documents in Medline and for the purpose of performing machine learning on sets of documents. The system has a number of unique features: 1) It is based on a number of C++ classes and highly modular so that alterations in the system are relatively simple to perform. 2) The system currently processes PubMed data by extracting from the Sybase repositories using a C++ interface to Sybase. However, a change in the interface portion of the system would allow it to be applied to any large database consisting of discrete textual records. 3) Data processed by the system is stored as compressed file structures, etc. These structures are updatable so that new data may be continually added to the system as it becomes available. 4) Documents are compared with each other using a Bayesian form of analysis. 5) Code has been multithreaded and memory mapping capabilities added to speed up processing. 6) Most recently the code has been updated to work in a 64 bit environment. The system described here is now not only being used to process all of MEDLINE for our research purposes, but also to produce the related documents for arbitrary pieces of text by other groups here in the NLM and outside of the NLM. The system is currently proving useful in testing different retrieval parameters and methods on the PubMedHealth records. We have recently developed a software system called DStor that allows us to store all of PubMed in a manner which is easily updateable and allows fast access. This system is now being used to maintain and update five different versions of the PubMed data twice a week. This system has greatly improved our access to PubMed data in various useful forms and we anctipate that its use will continue to grow. In addition we have developed software to maintain and update a list of strings where each string is associated with some fixed vector of integers. We currently maintain a list of all multi-word phrases without stop words or punctuation and with each is associated a vector of six integers representing counts of different types associated with each phrase where counts are computed over all PubMed records having abstracts. We also maintain a list of all one and two word phrases and MeSH terms in various forms (with & without stars and subheadings) and two counts with each consisting of the document frequency and the total frequency counting all occurrences in each document over all of PubMed.

为了在Medline中找到密切相关的文档，并为了在文档集上执行机器学习的目的，开发了一个c++语言程序系统。该系统有许多独特的特点：1)它基于许多c++类，并且高度模块化，因此对系统的更改相对容易执行。2)系统目前处理PubMed数据的方式是从Sybase存储库中提取数据，使用一个c++的Sybase接口。但是，对系统的接口部分进行更改将使其能够应用于由离散文本记录组成的任何大型数据库。3)系统处理的数据以压缩文件结构等形式存储。这些结构是可更新的，因此新数据可以在可用时不断添加到系统中。4)使用贝叶斯分析形式对文件进行相互比较。5)代码已被多线程和内存映射功能，以加快处理速度。6)最近的代码已经更新到在64位环境下工作。