RETRIEVAL TASK--COMPARE GROUP/INDIVIDUAL PERFORMANCE--SUBJECT EXPERTS/UNTRAINED

有用的检索取决于预测用户在回答查询时会发现哪些文档有帮助的能力。我们感兴趣的是常见的情况下，除了查询和查询是在自然语言中的用户提供任何信息。在这种情况下，人们普遍认为，一个人可以以判断的形式做出有用的预测，判断什么可能对另一个人有用。我们目前的数据表明，当一组人的预测平均，结果是一个更好的预测。如果以精度来衡量性能，则组性能随着组的大小而增加，并且接近我们数据上平均个人性能提高约50%的极限。团体的上级表现提出了如何实现的问题。我们研究的群体都是学科专家，一个自然的问题是，上级表现是否来自于他们学科知识的汇集。为了回答这个问题，我们还研究了一组未经训练的人。令我们惊讶的是，我们发现，虽然未经训练的个体的表现略逊于受过训练的个体，但未经训练的个体组合在一起的表现比任何一个受过训练的个体都要好，而且几乎达到了受过训练的群体的水平。发表了几篇文章。

Useful retrieval depends on the ability to predict which documents a user will find helpful in answer to a query. Our interest is the common case when no information is provided about the user other than the query and the query is in natural language. In this setting it is well accepted that a human can make useful predictions in the form of judgments about what will likely prove useful to another human. We present data showing that when the predictions of a group of humans are averaged, the result is a better predictor. If performance is measured as a precision, the group performance increases with the size of the group and approaches a limit of approximately 50% improvement over average individual performance on our data. Superior performance by groups raises the question of how. The groups we studied were subject experts and a natural question was whether the superior performance resulted from the pooling of their subject knowledge. In order to answer this question we studied also a group of untrained individuals. To our surprise we found that while untrained individuals had a somewhat inferior performance to trained individuals, the group of untrained individuals together performed better than any single trained individual and almost at the level of the trained group. Several articles have been published.