RETRIEVAL TASK--COMPARE GROUP/INDIVIDUAL PERFORMANCE--SUBJECT EXPERTS/UNTRAINED

有用的检索依赖于预测用户在回答查询时会发现哪些文档有用的能力。我们感兴趣的是常见的情况，即除了查询之外没有提供关于用户的任何信息，并且查询是用自然语言进行的。在这种情况下，人们普遍认为，人类可以通过判断的形式做出有用的预测，判断什么可能对另一个人有用。我们提供的数据表明，当一群人的预测是平均的，结果是一个更好的预测。如果性能以精度来衡量，则组性能随着组的大小而增加，并且接近于我们数据上的平均个人性能提高的大约50%的极限。团队的卓越表现提出了一个问题：如何做到这一点？我们研究的小组是学科专家，一个自然的问题是，卓越的表现是否源于他们学科知识的汇集。为了回答这个问题，我们还研究了一组未经训练的人。令我们惊讶的是，我们发现，虽然未经训练的人的表现略逊于受过训练的人，但未经训练的人加在一起的表现比任何一个受过训练的人都要好，几乎达到了受过训练的人的水平。已经发表了几篇文章。-个人表现，团体表现，未经培训的工人，主题专家-人类受试者：采访，问卷调查或调查

Useful retrieval depends on the ability to predict which documents a user will find helpful in answer to a query. Our interest is the common case when no information is provided about the user other than the query and the query is in natural language. In this setting it is well accepted that a human can make useful predictions in the form of judgments about what will likely prove useful to another human. We present data showing that when the predictions of a group of humans are averaged, the result is a better predictor. If performance is measured as a precision, the group performance increases with the size of the group and approaches a limit ofapproximately50% improvement over average individual performance on our data. Superior performance by groups raises the question of how. The groups we studied were subject experts and a natural question was whether the superior performance resulted from the pooling of their subject knowledge. In order to answer this question we studied also a group of untrained individuals. To our surprise we found that while untrained individuals had a somewhat inferior performance to trained individuals, the group of untrained individuals together performed better than any single trained individual and almost at the level of the trained group. Several articles have been published. - individual performance, group performance, untrained workers, subject experts - Human Subjects: Interview, Questionaires, or Surveys Only