Theme Recognition In Document Collections

人们已经研究了许多不同的方法来对文档集进行聚类，以期提高检索效率。不幸的是，这些通常不能提供改进的检索能力。问题的一部分显然是这样一个事实，即一份特定文件往往涉及一个以上的主题，因此不可能将文件明确地分类为确定的类别，而不包括其他类别。为了克服这一问题，我们开发了旨在从一组文件中确定主题的方法。主题不必涵盖任何文件的全部内容。它只需要存在于文档的某个子集中即可识别。这些文件中的一些文件可能参与了若干主题的定义。发现主题的方法基于EM算法，并且需要收敛到主题的迭代过程。该方法已被实施和测试，证明是成功的。最近，该方法得到了改进，我们已将其应用于检测数据库中的主题，该数据库包含52000份关于艾滋病主题的文件和200多万份涉及淋巴细胞主题的文件。最近，我们完成了将其应用于所有MEDLINE以定义107,863个主题。目前正在对这些数据库进行测试，以了解它们如何用于加强MEDLINE和可能的其他数据库的检索。我们在这个项目上的最新工作是开发一种方法，以产生一个相互链接的术语网络，因为它们彼此相关。这起到了取代个别主题的作用。我们正在试验这种网络，以此作为增强检索的一种方式。

Many different methods have been investigated for the purpose of clustering sets of documents with the hope of improving retrieval. Unfortunately these have generally failed to provide improved retrieval capability. Part of the problem is clearly the fact that a given document often involves more than one subject so that it is not possible to make a clean categorization of the documents into definite categories to the exclusion of others. In order to overcome this problem we have developed methods that are designed to identify a theme among a set of documents. The theme need not encompass the whole of any document. It only needs to exist in some subset of the documents in order to be identifiable. Some of these same documents may participate in the definition of several themes. The method of finding themes is based on the EM algorithm and requires an iterative procedure which converges to themes. The method has been implemented and tested and found to be successful. Recently the method has been improved and we have applied it to detect themes in a database of 52k documents on the subject of AIDS and over 200k documents dealing with the subject of lymphocytes. Recently we completed applying it to all of MEDLINE to define 107,863 themes. These are now being tested to see how they may be used to enhance retrieval in MEDLINE and possibly other databases. Our most recent work on this project is to develop an approach to produce a network of terms that are linked to each other as they are relevant to each other. This serves to take the place of individual themes. We are experimenting with this network as a way to enhance retrieval.