Repeatable and reliable search system evaluation using crowdsourcing
Repeatable and reliable search system evaluation using crowdsourcing
复制标题
DOI：
10.1145/2009916.2010039
复制
发表时间：
2011-07
期刊：
影响因子：
--
通讯作者：
Roi Blanco;H. Halpin;Daniel M. Herzig;P. Mika;Jeffrey Pound;H. Thompson;Thanh Tran
中科院分区：
文献类型：
--
作者：
Roi Blanco;H. Halpin;Daniel M. Herzig;P. Mika;Jeffrey Pound;H. Thompson;Thanh Tran
任何一种新的搜索任务面临的主要问题是如何引导一个可靠和可重复的评估活动，而众包方法提供了许多优势。然而，这些众包评估能否长期可靠地重复进行？为了说明这一点，我们研究了为基于关键字的即席对象检索的语义搜索任务创建评估活动。与传统的网页搜索不同，对象搜索的目标是从关于真实世界对象的事实断言中检索信息，而不是在带有文本描述的网页上进行搜索。使用第一个专门针对多个已部署系统上的特别Web对象检索任务的大规模评估活动，我们证明了众包评估活动可以随着时间的推移重复进行，并且仍然保持可靠的结果。此外，我们展示了当对系统进行排名时，这些结果如何与专家评委相比较，并且这些结果在不同的评估和相关性度量上是成立的。这项工作为使用众包进行可扩展、可靠和可重复的搜索系统评估提供了经验支持。
The primary problem confronting any new kind of search task is how to boot-strap a reliable and repeatable evaluation campaign, and a crowd-sourcing approach provides many advantages. However, can these crowd-sourced evaluations be repeated over long periods of time in a reliable manner? To demonstrate, we investigate creating an evaluation campaign for the semantic search task of keyword-based ad-hoc object retrieval. In contrast to traditional search over web-pages, object search aims at the retrieval of information from factual assertions about real-world objects rather than searching over web-pages with textual descriptions. Using the first large-scale evaluation campaign that specifically targets the task of ad-hoc Web object retrieval over a number of deployed systems, we demonstrate that crowd-sourced evaluation campaigns can be repeated over time and still maintain reliable results. Furthermore, we show how these results are comparable to expert judges when ranking systems and that the results hold over different evaluation and relevance metrics. This work provides empirical support for scalable, reliable, and repeatable search system evaluation using crowdsourcing.