A Document Processing System

一个系统的C++语言程序已经开发的目的是找到密切相关的文件在Medline和执行机器学习的文件集的目的。该系统具有许多独特的功能：1）它基于许多C++类和高度模块化，因此系统中的更改相对简单。2)该系统目前通过使用C++接口从PubMed数据库中提取数据来处理PubMed数据。但是，系统接口部分的变化将使其适用于由离散文本记录组成的任何大型数据库。3)由系统处理的数据被存储为压缩文件结构等。这些结构是可更新的，使得新数据可以在其变得可用时被持续地添加到系统。4)使用贝叶斯分析形式对文档进行相互比较。5)代码已被多线程化，并添加了内存映射功能以加快处理速度。6)最近，代码已更新为在64位环境中工作。 这里描述的系统现在不仅用于处理我们研究目的的所有MEDLINE，而且还用于为NLM内外的其他小组的任意文本生成相关文档。该系统目前被证明是有用的，在测试不同的检索参数和方法的PubMedHealth记录。 我们最近开发了一个软件系统，名为Docs，使我们能够存储所有的PubMed的方式，这是很容易更新，并允许快速访问。该系统现在用于每周两次维护和更新五个不同版本的PubMed数据。该系统极大地改善了我们对各种有用形式的PubMed数据的访问，我们认为它的使用将继续增长。此外，我们还开发了软件来维护和更新一个字符串列表，其中每个字符串都与某个固定的整数向量相关联。我们目前维护了一个没有停用词或标点符号的所有多词短语的列表，每个短语都与一个六个整数的向量相关联，该向量表示与每个短语相关联的不同类型的计数，其中计数是在具有摘要的所有PubMed记录上计算的。我们还维护了一个列表，其中包含各种形式的所有一个和两个单词短语以及MeSH术语（带和不带星号和副标题），以及两个计数，每个计数包括文档频率和总频率，每个文档中的所有出现次数都超过了PubMed。

A system of C++ language programs has been developed for the purpose of finding the closely related documents in Medline and for the purpose of performing machine learning on sets of documents. The system has a number of unique features: 1) It is based on a number of C++ classes and highly modular so that alterations in the system are relatively simple to perform. 2) The system currently processes PubMed data by extracting from the Sybase repositories using a C++ interface to Sybase. However, a change in the interface portion of the system would allow it to be applied to any large database consisting of discrete textual records. 3) Data processed by the system is stored as compressed file structures, etc. These structures are updatable so that new data may be continually added to the system as it becomes available. 4) Documents are compared with each other using a Bayesian form of analysis. 5) Code has been multithreaded and memory mapping capabilities added to speed up processing. 6) Most recently the code has been updated to work in a 64 bit environment. The system described here is now not only being used to process all of MEDLINE for our research purposes, but also to produce the related documents for arbitrary pieces of text by other groups here in the NLM and outside of the NLM. The system is currently proving useful in testing different retrieval parameters and methods on the PubMedHealth records. We have recently developed a software system called DStor that allows us to store all of PubMed in a manner which is easily updateable and allows fast access. This system is now being used to maintain and update five different versions of the PubMed data twice a week. This system has greatly improved our access to PubMed data in various useful forms and we anctipate that its use will continue to grow. In addition we have developed software to maintain and update a list of strings where each string is associated with some fixed vector of integers. We currently maintain a list of all multi-word phrases without stop words or punctuation and with each is associated a vector of six integers representing counts of different types associated with each phrase where counts are computed over all PubMed records having abstracts. We also maintain a list of all one and two word phrases and MeSH terms in various forms (with & without stars and subheadings) and two counts with each consisting of the document frequency and the total frequency counting all occurrences in each document over all of PubMed.