A Document Processing System

开发了一个C++语言程序系统，目的是在Medline中查找密切相关的文件，并对文件集进行机器学习。该系统具有许多独特的功能：1)它基于一些C++类，并且高度模块化，因此系统中的更改相对容易执行。2)系统目前通过使用Sybase的C++接口从Sybase存储库中提取PubMed数据来处理。但是，该系统界面部分的改变将使其适用于由离散文本记录组成的任何大型数据库。3)由系统处理的数据被存储为压缩文件结构等。这些结构是可更新的，从而当新数据变得可用时，可以不断地向系统添加新数据。4)使用贝叶斯分析形式对文档进行相互比较。5)关于该系统的最新工作包括添加使用EM算法方法生成主题的能力。此外，最近还增加了多线程代码和内存映射功能，以加快处理速度。 这里描述的系统现在不仅被用于处理所有MEDLINE用于我们的研究目的，而且还被用于为NLM内部和外部的其他小组产生任意文本片段的相关文档。该系统已用于挖掘NLM帮助台的电子邮件通信。

A system of C++ language programs has been developed for the purpose of finding the closely related documents in Medline and for the purpose of performing machine learning on sets of documents. The system has a number of unique features: 1) It is based on a number of C++ classes and highly modular so that alterations in the system are relatively simple to perform. 2) The system currently processes PubMed data by extracting from the Sybase repositories using a C++ interface to Sybase. However, a change in the interface portion of the system would allow it to be applied to any large database consisting of discrete textual records. 3) Data processed by the system is stored as compressed file structures, etc. These structures are updatable so that new data may be continually added to the system as it becomes available. 4) Documents are compared with each other using a Bayesian form of analysis. 5) The latest work on this system has involved adding the ability to generate themes using an EM algorithm approach. Also recently code has been multithreaded and memory mapping capabilities added to speed up processing. The system described here is now not only being used to process all of MEDLINE for our research purposes, but also to produce the related documents for arbitrary pieces of text by other groups here in the NLM and outside of the NLM. The system has been used for mining email communications for the NLM help desk.