EAGER: A Nugget-Based Information Retrieval Evaluation Paradigm

评价信息检索系统，如搜索引擎，是其有效发展的关键。当前的性能评估方法通常是克兰菲尔德范例的变体，该范例依赖于有效地完整的、因此昂贵得令人望而却步的相关性判断集：必须由人类评估员判断数万到数十万个文档与数十到数百个用户查询的相关性，这在时间和费用上都是巨大的成本。本研究探讨一种新的资讯撷取评估模式--以“金块”为基础。 “金块”是相关信息的原子单位，这些金块的一个实例是简单的句子或简短的段落，导致法官在评估文件时认为文件相关。假设是，虽然可能不可能找到关于网络规模和/或动态集合的查询的所有相关文档，但是找到所有或几乎所有的金块（即，相关信息），然后可以大规模轻松地进行有效和可重复使用的评估。在评估时，基于在检索到的文档中发现的相关信息的数量和质量，动态地为文档创建相关性评估。这种新的评价范式本质上是可扩展的，并允许使用所有标准的检索性能的措施，包括那些涉及分级相关性判断，新奇，多样性，等等;它还允许新的类型的评价迄今为止不可能的。在促进这一努力的过程中，该项目小组与美国国家标准与技术研究所（NIST）和日本国家信息学研究所（通过NTCIR）有着密切的联系，这两个机构是开发和发布信息检索数据集的主要组织。作为该项目一部分而开发的所有研究成果和数据集均可在项目网站上查阅（http：//www.ccs.neu.edu/home/jaa/IIS-1256172/）。该项目还为学生提供教育和培训经验，并根据项目结果编写教材。

Evaluating information retrieval systems, such as search engines, is critical to their effective development. Current performance evaluation methodologies are generally variants of the Cranfield paradigm, which relies on effectively complete, and thus prohibitively expensive, relevance judgment sets: tens to hundreds of thousands of documents must be judged by human assessors for relevance with respect to dozens to hundreds of user queries, at great cost both in time and expense. This exploratory project investigates a new alternative to information retrieval evaluation paradigm -- based on "nuggets". "Nuggets" are atomic units of relevant information, and one instantiation of these nuggets is simply the sentence or short passage that causes a judge to deem a document relevant at the time of document assessment. The hypothesis is that while it is likely impossible to find all relevant documents for a query with respect to web-scale and/or dynamic collections, it is much more tractable to find all or nearly all nuggets (i.e., relevant information), with which one can then perform effective and reusable evaluation, at scale and with ease. At evaluation time, relevance assessments are dynamically created for documents based on the quantity and quality of relevant information found in the documents retrieved. This new evaluation paradigm is inherently scalable and permits the use of all standard measures of retrieval performance, including those involving graded relevance judgments, novelty, diversity, and so on; it further permits new kinds of evaluations not heretofore possible.The project plan includes the development and release of nugget-based evaluation data sets for use by academia and industry. In fostering this effort, the project team has close ties with the US National Institute of Standards and Technology (NIST) and the Japanese National Institute of Informatics (through NTCIR), two of the premier organizations that develop and release information retrieval data sets. All research results and data sets developed as part of this project are available at the project website (http://www.ccs.neu.edu/home/jaa/IIS-1256172/). The project also provides educational and training experience for students and the development of curricular materials based on the project results.