III: Small: Optimal Allocation of Crowdsourced Resources for IR Evaluation

评价搜索引擎等信息检索系统的性能对于它们的有效发展至关重要。目前的“黄金标准”绩效评估方法一般依赖于使用专家评估员来判断搜索引擎检索到的文件或网页的质量，这在时间和费用上都付出了很大的代价。 “众包”的出现，例如通过亚马逊的土耳其机器人服务提供的服务，提供了这样的希望，即通过使用许多（尽管通常技能较低）“众包工作者”，这些绩效评估可以更快地进行，成本低得多;然而，最终绩效评估的质量通常会受到很大影响。 这个项目的论点是，一个人可以获得两全其美--以专家的质量进行绩效评估，但以群众工作者为代价--通过最佳地利用专家和群众工作者在“正确”的时间向“正确”的评估者提出“正确”的问题。 例如，人们可能会问廉价的群众工作者什么可能是“简单”的问题，而把什么可能是“困难”的问题留给昂贵的专家。 虽然该项目侧重于搜索引擎的性能评估作为其用例，但开发的技术将更广泛地适用于许多领域，在这些领域中，人们希望有效地利用具有不同成本和专业知识水平的专家和人群工作者。我们将建立一个概率框架，在这个框架内，我们可以量化绩效评估的不确定性，以及要求任何评估人员的成本和预期效用（专家或群众工作者）任何问题（例如，对文档的名义判断或两个文档之间的偏好判断）。 然后，目标是在任何时候向“正确”的评估者提出“正确”的问题，以最大限度地提高单位成本的预期效用，然后最佳地汇总这些回答，以有效地评估绩效。 欲了解更多信息，请访问项目网站：http://www.ccs.neu.edu/home/jaa/IIS-1421399/。

Evaluating the performance of information retrieval systems such as search engines is critical to their effective development. Current "gold standard" performance evaluation methodologies generally rely on the use of expert assessors to judge the quality of documents or web pages retrieved by search engines, at great cost in time and expense. The advent of "crowd sourcing," such as available through Amazon's Mechanical Turk service, holds out the promise that these performance evaluations can be performed more rapidly and at far less cost through the use of many (though generally less skilled) "crowd workers"; however, the quality of the resulting performance evaluations generally suffer greatly. The thesis of this project is that one can obtain the best of both worlds -- performance evaluations with the quality of experts but at the cost of crowd workers -- by optimally leveraging both experts and crowd workers in asking the "right" assessor the "right" question at the "right" time. For example, one might ask inexpensive crowd workers what are likely to be "easy" questions while reserving what are likely to be "hard" questions for the expensive experts. While the project focuses on the performance evaluation of search engines as its use case, the techniques developed will be more broadly applicable to many domains where one wishes to efficiently and effectively harness experts and crowd workers with disparate levels of cost and expertise.To enable the vision described above, a probabilistic framework will be developed within which one can quantify the uncertainty about a performance evaluation as well as the cost and expected utility of asking any assessor (expert or crowd worker) any question (e.g. a nominal judgment for a document or a preference judgment between two documents) at any time. The goal is then to ask the "right" question of the "right" assessor at any time in order to maximize the expected utility gained per unit cost incurred and then to optimally aggregate such responses in order to efficiently and effectively evaluate performance. For further information, see the project website at: http://www.ccs.neu.edu/home/jaa/IIS-1421399/.