TEXTUAL INFORMATION RETRIEVAL TESTING

检索测试是困难的，原因有几个。 第一， 进行实际检索的文件大到 一点是，创建完整的人类判断的测试集不是 可能 正因为如此，我们刚刚完成了一项大型研究， 在分子生物学领域的收藏。 对于这个大型数据库， 检索操作进行了模拟和性能的相关措施， 与绝对值相比。 模拟是基于 McCarn-Lewis模型方程 已经发现， 正如我们所构建的那样，衡量标准是一个敏感的绝对指标， 性能，但一个稍微偏离绝对措施， 在某些情况下可以很容易地预测的系统方法 所测试的检索方法的可测量属性。 检索测试困难的第二个因素是缺乏 实际检索性能的模型。 这使得难以 评估测试时获得的结果的重要性。 一 必须使用关于测试的对称性的特殊假设 通过不同的检索方法获得的结果等。这些假设 很少满足。 因此，我们研究了bootstrap 方法. 基于这些结果，我们得出结论， 方法提供了一种可靠实用的显著性检验方法 检索性能结果。 第三个使检索测试困难的方面是缺乏 人类相关性判断的再现性。 我们采取的方法是， 因此，相关性必须被理解为一种可能性。 这允许 概率排序原理的独特解释。 我们的研究 表明这种解释提供了一种稳定的测试方法 这限制了性能，并允许比较人类和 机器性能平等

Retrieval testing is difficult for several reasons. First, the sets of documents on which actual retrieval are carried out are large to the point that the creation of complete humanly judged test sets is not possible. Because of this we have just completed a study of a large collection in the area of molecular biology. For this large database the retrieval operation is simulated and relative measures of performance are compared with absolute measures. The simulations are based on the McCarn-Lewis model equation. It has been found that the relative measure, as we have constructed it, is a sensitive indicator of absolute performance but one which deviates slightly from the absolute measure in a systematic way which can in some circumstances be predicted from easily measurable properties of the retrieval method being tested. The second factor that makes retrieval testing difficult is the lack of a model for actual retrieval performance. This makes it difficult to assess the significance of results obtained when testing is done. One must use special assumptions about the symmetry properties of test results obtained by different retrieval methods, etc. Such assumptions are seldom satisfied. We have therefore investigated the bootstrap methods. Based on these results we have concluded that the bootstrap methods provide a reliable and practical method of significance testing for retrieval performance results. The third area that makes retrieval testing difficult is the lack of reproducibility of human relevance judgements. We take the approach that relevance must therefore be understood as a probability. This allows a unique interpretation of the probability ranking principle. Our studies show that such an interpretation provides a stable approach to testing that places limits on performance and allows the comparison of human and machine performance on an equal footing