AUTOMATIC BAYESIAN METHODS IN TEXT RETRIEVAL

一种自动化的方法，应用贝叶斯方法在文本检索， 开发了这是一种搜索词的相关性加权形式 但在两个方面与通常的方法不同， 其他.首先，通常的方法涉及关联的分配 基于文档的单个查询中搜索词的权重 和那些与查询不相关的。这涉及 一般来说，只有少量的相关文件，因此， 这是一个很难使用的样本， 关于所涉及的术语的值的推断。我们修改通常的 方法通过取一个术语在所有 查询它发生的地方。我们研究的情况下，当一组查询是 使得全局术语相关性权重是一个很好的 定义的概念。第二，通常的方法仅限于以下情况： 人们对文档与查询的相关性进行人工判断。这 将该方法的使用限制在某些测试集， 相关性关系是已知的或者与相关性反馈情况相关。我们 方法是用高关联关系代替关联关系 使用向量余弦方法对查询和文档对进行评分， 检索 由于后者是一种自动方法，我们能够 以自动方式生成所需的统计数据。虽然这 后一种方法无疑会比人的相关性有更多的错误 全球加权所涉及的较大样本量有助于 抵消这个问题。 局部加权是以特别的方式引入的 结果表明，该方法的检索效果上级矢量检索 余弦反演 上述模型存在两个问题。首先， 以自然贝叶斯方式结合局部项加权， 它不提供对文档长度的校正。我们已经开发 一种基于集群概念的新模型，可以解决这两个问题 同时允许模型保持完全贝叶斯。这表现在 与已经描述的相同的基本水平，其中局部权重 都是特别处理的然而，它允许人们看到实际的日志赔率 相关性的预测。这些超过了观察到的 13：1的相关性，这给出了一个有趣的观点， 依赖

An automatic approach to applying Bayesian methods in text retrieval has been developed. This is a form of relevance weighting of search terms but departs from the usual approach in two ways which complement each other. First, the usual approach involves the assignment of relevance weights to the search terms in a single query based on the documents that are and those that are not relevant to the query. This involves generally a small number of relevant documents and hence a statistical sample that is difficult to use in making any globally significant inferences about the value of the terms involved. We modify the usual approach by taking the average of the importance of a term over all the queries in which it occurs. We study the case when the set of queries is the set of documents so that the global term relevance weight is a well defined concept. Second, the usual approach is limited to the case when one has human judgments of the relevance of documents to queries. This has limited the use of the method to certain test sets where the relevance relation is known or to relevance feedback situations. Our approach is to replace the relevance relation by the relation of high scoring pairs of query and document using the vector cosine method of retrieval. Because the latter is an automatic method we are able to generate the required statistics in an automatic manner. While this latter approach will undoubtedly have more error than human relevance judgments the larger sample size involved in global weighting helps to offset this problem. Local weighting is introduced in an ad hoc manner and the resultant retrieval is found to be somewhat superior to vector cosine retrieval. There are two problems with the model just described. First it does not incorporate local term weighting in a natural Bayesian manner and second it does not provide a correction for document length. We have developed a new model based on cluster concepts that remedies these two problems while allowing the model to remain completely Bayesian. This performs at the same basic level as the one already described in which local weights are treated ad hoc. It does however allow one to see the actual log odds predictions of relevance. These exceed the observed log odds of relevance by 13:1 which gives an interesting perspective on term dependency.