A DOCUMENT PROCESSING SYSTEM

已经开发了一个软件系统，目的是寻找 Medline中密切相关的文件。该系统由多个 35个用C语言编写的程序以及一些实用程序 程序。该系统具有许多独特的功能： 1)它是高度模块化的，因此系统中的更改相对 执行起来很简单。 2)该系统目前处理的是ASN1格式的Medline数据，但 系统界面部分的更改将使其能够 适用于任何由离散文本记录组成的大型数据库。 3)系统设计具有一定的防数据丢失的安全性 由于操作系统崩溃或停电。 4)系统处理的所有数据都以永久形式存储为 倒置文件结构等。这些结构是可更新的，以便 当新数据变得可用时，可以不断地将其添加到系统中。 5)文档之间使用贝叶斯形式进行比较 词语关联度权重的分析和统计 基于的是从以前的文档比较中派生出来的。这些统计数据 随着每个新的处理周期而更新。 6)文档相关的概率由系统计算 基于使用一组文档产生的原始分数的比例 已经被人类评委判定为关联性的配对。这个比例尺 每次更新术语权重时都会重新计算，并进行计算 具有的文档与不具有的文档不同 摘要。 对系统中最明显的故障进行分析，如 实现了用于文档相似度度量的测试集 出去。这表明，所经历的问题中有很大一部分可能 是由于几个不同级别的地区共同发生的 中的描述以及描述中不同级别的一致性 单个文档。这些描述性区域的数字表示 通常与它们在文档中的重要性无关 定义文档内容。

A system of software has been developed for the purpose of finding the closely related documents in Medline. The system consists of over thirty-five programs written in the C language plus a number of utility programs. The system has a number of unique features: 1) It is highly modular so that alterations in the system are relatively simple to perform. 2) The system currently operates on Medline data in the ASN1 format but a change in the interface portion of the system would allow it to be applied to any large database consisting of discrete textual records. 3) The system is designed with a degree of security against loss of data due to operating system crashes or power outages. 4) All data processed by the system is stored in permanent form as inverted file structures, etc. These structures are updateable so that new data may be continually added to the system as it becomes available. 5) Documents are compared with each other using a Bayesian form of analysis and the statistics on which the relevance weighting of terms is based are derived from previous document comparisons. These statistics are updated with each new cycle of processing. 6) The probability that documents are related is computed by the system based on a scaling of the raw scores produced using a set of document pairs that have been judged for relatedness by human judges. This scale is recalculated each time term weights are updated and it is calculated differently for documents with as opposed to documents without abstracts. An analysis of the most glaring failures of the system, as identified on the test set used for scaling of document similarity, has been carried out. This shows that a significant part of the problems experienced may be due to the common occurrence of several areas with different levels of description and different levels of uniformity in description in a single document. The numerical representation of these descriptive areas within a document does not in general correlate with their importance in defining document content.