RETRIEVAL TASK--COMPARE GROUP/INDIVIDUAL PERFORMANCE--SUBJECT EXPERTS/UNTRAINED

有用的检索取决于预测的能力 用户会发现这些文档对回答查询有帮助。我们的 当没有提供有关的信息时，利息是常见的情况 除了查询和查询之外的用户是自然语言。 在这种情况下，人们普遍认为，人类可以使 对可能会证明什么的判断形式的预测 对另一个人类有用。我们提供的数据表明，当 对一组人的预测进行平均，结果是更好的 预言者。如果绩效是以精确度衡量的，那么团队 性能随着组的规模增加而提高，并接近 比平均个人提高约50%的限制 对我们的数据的性能。团队的卓越表现提升了 如何做到的问题。我们研究的小组都是学科专家 一个自然的问题是，卓越的表现是否 是他们的学科知识汇聚的结果。为了 回答这个问题我们还研究了一群未经培训的 个人。令我们惊讶的是，我们发现在未经训练的情况下 个人的表现略逊于训练有素的 个人，一群未受过训练的个人一起表演 比任何一个训练有素的人都要好，几乎达到了 训练有素的团队。已经发表了一篇文章。

Useful retrieval depends on the ability to predict which documents a user will find helpful in answer to a query. Our interest is the common case when no information is provided about the user other than the query and the query is in natural language. In this setting it is well accepted that a human can make useful predictions in the form of judgments about what will likely prove useful to another human. We present data showing that when the predictions of a group of humans are averaged, the result is a better predictor. If performance is measured as a precision, the group performance increases with the size of the group and approaches a limit of approximately 50% improvement over average individual performance on our data. Superior performance by groups raises the question of how. The groups we studied were subject experts and a natural question was whether the superior performance resulted from the pooling of their subject knowledge. In order to answer this question we studied also a group of untrained individuals. To our surprise we found that while untrained individuals had a somewhat inferior performance to trained individuals, the group of untrained individuals together performed better than any single trained individual and almost at the level of the trained group. An article has been published.