III: Small: Optimal Allocation of Crowdsourced Resources for IR Evaluation

III：小：IR评估众包资源的优化配置

• 批准号: 1421399
• 负责人: Javed Aslam
• 金额: $ 49.97万
• 依托单位: Northeastern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-08-15 至 2017-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1421399&HistoricalAwards=false
• 关键词: III; Small; Optimal; Allocation; Crowdsourced

Evaluating the performance of information retrieval systems such as search engines is critical to their effective development. Current "gold standard" performance evaluation methodologies generally rely on the use of expert assessors to judge the quality of documents or web pages retrieved by search engines, at great cost in time and expense. The advent of "crowd sourcing," such as available through Amazon's Mechanical Turk service, holds out the promise that these performance evaluations can be performed more rapidly and at far less cost through the use of many (though generally less skilled) "crowd workers"; however, the quality of the resulting performance evaluations generally suffer greatly. The thesis of this project is that one can obtain the best of both worlds -- performance evaluations with the quality of experts but at the cost of crowd workers -- by optimally leveraging both experts and crowd workers in asking the "right" assessor the "right" question at the "right" time. For example, one might ask inexpensive crowd workers what are likely to be "easy" questions while reserving what are likely to be "hard" questions for the expensive experts. While the project focuses on the performance evaluation of search engines as its use case, the techniques developed will be more broadly applicable to many domains where one wishes to efficiently and effectively harness experts and crowd workers with disparate levels of cost and expertise.To enable the vision described above, a probabilistic framework will be developed within which one can quantify the uncertainty about a performance evaluation as well as the cost and expected utility of asking any assessor (expert or crowd worker) any question (e.g. a nominal judgment for a document or a preference judgment between two documents) at any time. The goal is then to ask the "right" question of the "right" assessor at any time in order to maximize the expected utility gained per unit cost incurred and then to optimally aggregate such responses in order to efficiently and effectively evaluate performance. For further information, see the project website at: http://www.ccs.neu.edu/home/jaa/IIS-1421399/.

评估搜索引擎等信息检索系统的性能对于其有效开发至关重要。目前的“黄金标准”绩效评估方法通常依赖于使用专家评估员来判断搜索引擎检索到的文档或网页的质量，这会耗费大量的时间和费用。 “众包”的出现，例如通过亚马逊的 Mechanical Turk 服务提供的服务，承诺通过使用许多（尽管通常技能较低）“众包工人”，可以更快地执行这些绩效评估，并且成本要低得多；然而，绩效评估结果的质量通常会受到很大影响。 该项目的主题是，通过最佳地利用专家和众包工作者在“正确”的时间向“正确”的评估者提出“正确”的问题，可以获得两全其美的效果——以专家的质量进行绩效评估，但以众包工作者的成本为代价。 例如，人们可能会问廉价的众包工作者哪些可能是“简单”的问题，而向昂贵的专家保留哪些可能是“困难”的问题。 虽然该项目侧重于将搜索引擎的性能评估作为其用例，但所开发的技术将更广泛地适用于许多领域，在这些领域中，人们希望高效、有效地利用具有不同成本和专业知识水平的专家和群众工作者。为了实现上述愿景，将开发一个概率框架，在该框架中，人们可以量化性能评估的不确定性以及询问任何评估员（专家或群众工作者）的成本和预期效用 任何时候的任何问题（例如，对一份文件的名义判断或对两份文件的偏好判断）。 然后，目标是随时向“正确”评估者提出“正确”问题，以便最大化每单位成本所获得的预期效用，然后以最佳方式聚合此类响应，以便高效且有效地评估绩效。 有关更多信息，请参阅项目网站：http://www.ccs.neu.edu/home/jaa/IIS-1421399/。