EAGER: A Nugget-Based Information Retrieval Evaluation Paradigm

评估信息检索系统，如搜索引擎，是其有效发展的关键。当前的性能评估方法通常是克兰菲尔德范式的变体，它依赖于有效完整的相关性判断集，因此非常昂贵：成千上万的文档必须由人工评估人员对数十到数百个用户查询进行相关性判断，这在时间和费用上都付出了巨大的代价。这个探索性项目研究了一种新的替代信息检索评估范式——基于“掘金”。“掘金”是相关信息的原子单位，这些掘金的一个实例就是在文件评估时导致法官认为文件相关的句子或简短段落。假设是，虽然不可能找到与web规模和/或动态集合相关的所有相关文档，但找到所有或几乎所有的金块（即相关信息）要容易得多，然后可以在规模和轻松的情况下执行有效和可重用的评估。在评估时，根据检索到的文档中找到的相关信息的数量和质量，动态地为文档创建相关性评估。这种新的评估范式具有固有的可扩展性，并允许使用检索性能的所有标准度量，包括涉及分级相关性判断、新颖性、多样性等的标准度量；它进一步使以前不可能的新评价成为可能。该项目计划包括开发和发布基于金块的评估数据集，供学术界和工业界使用。为了促进这一努力，项目团队与美国国家标准与技术研究所（NIST）和日本国家信息学研究所（通过NTCIR）建立了密切的联系，这两个机构是开发和发布信息检索数据集的主要组织。作为该项目的一部分，所有研究结果和数据集可在项目网站（http://www.ccs.neu.edu/home/jaa/IIS-1256172/）上获得。该项目还为学生提供教育和培训经验，并根据项目成果开发课程材料。

Evaluating information retrieval systems, such as search engines, is critical to their effective development. Current performance evaluation methodologies are generally variants of the Cranfield paradigm, which relies on effectively complete, and thus prohibitively expensive, relevance judgment sets: tens to hundreds of thousands of documents must be judged by human assessors for relevance with respect to dozens to hundreds of user queries, at great cost both in time and expense. This exploratory project investigates a new alternative to information retrieval evaluation paradigm -- based on "nuggets". "Nuggets" are atomic units of relevant information, and one instantiation of these nuggets is simply the sentence or short passage that causes a judge to deem a document relevant at the time of document assessment. The hypothesis is that while it is likely impossible to find all relevant documents for a query with respect to web-scale and/or dynamic collections, it is much more tractable to find all or nearly all nuggets (i.e., relevant information), with which one can then perform effective and reusable evaluation, at scale and with ease. At evaluation time, relevance assessments are dynamically created for documents based on the quantity and quality of relevant information found in the documents retrieved. This new evaluation paradigm is inherently scalable and permits the use of all standard measures of retrieval performance, including those involving graded relevance judgments, novelty, diversity, and so on; it further permits new kinds of evaluations not heretofore possible.The project plan includes the development and release of nugget-based evaluation data sets for use by academia and industry. In fostering this effort, the project team has close ties with the US National Institute of Standards and Technology (NIST) and the Japanese National Institute of Informatics (through NTCIR), two of the premier organizations that develop and release information retrieval data sets. All research results and data sets developed as part of this project are available at the project website (http://www.ccs.neu.edu/home/jaa/IIS-1256172/). The project also provides educational and training experience for students and the development of curricular materials based on the project results.