Theme Recognition In Document Collections

为了提高检索效率，人们研究了许多不同的方法来聚类文档集。不幸的是，这些系统通常无法提供改进的检索能力。问题的一部分显然是这样一个事实，即一份给定的文件往往涉及一个以上的主题，因此不可能在排除其他主题的情况下将文件明确地分类为明确的类别。为了克服这个问题，我们开发了旨在从一组文档中确定主题的方法。主题不需要包括任何文件的全部内容。它只需要存在于文档的某个子集中就可以被识别。其中一些相同的文件可能参与几个主题的定义。寻找主题的方法是基于EM算法，需要一个迭代过程，该过程收敛于主题。该方法已被实施和测试，发现是成功的。最近，该方法得到了改进，我们已将其应用于检测数据库中有关艾滋病主题的5.2万份文件和涉及淋巴细胞主题的20多万份文件的主题。最近，我们完成了将其应用于所有MEDLINE以定义107,863个主题。目前正在对它们进行测试，以了解如何使用它们来增强MEDLINE和其他可能的数据库的检索。我们在这个项目上的最新工作是开发一种方法来生成一个相互关联的术语网络。这是为了取代单个主题。

Many different methods have been investigated for the purpose of clustering sets of documents with the hope of improving retrieval. Unfortunately these have generally failed to provide improved retrieval capability. Part of the problem is clearly the fact that a given document often involves more than one subject so that it is not possible to make a clean categorization of the documents into definite categories to the exclusion of others. In order to overcome this problem we have developed methods that are designed to identify a theme among a set of documents. The theme need not encompass the whole of any document. It only needs to exist in some subset of the documents in order to be identifiable. Some of these same documents may participate in the definition of several themes. The method of finding themes is based on the EM algorithm and requires an iterative procedure which converges to themes. The method has been implemented and tested and found to be successful. Recently the method has been improved and we have applied it to detect themes in a database of 52k documents on the subject of AIDS and over 200k documents dealing with the subject of lymphocytes. Recently we completed applying it to all of MEDLINE to define 107,863 themes. These are now being tested to see how they may be used to enhance retrieval in MEDLINE and possibly other databases. Our most recent work on this project is to develop an approach to produce a network of terms that are linked to each other as they are relevant to each other. This serves to take the place of individual themes.