EAGER: A Nugget-Based Information Retrieval Evaluation Paradigm

EAGER：基于块的信息检索评估范式

• 批准号: 1256172
• 负责人: Javed Aslam
• 金额: $ 15万
• 依托单位: Northeastern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-09-15 至 2015-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1256172&HistoricalAwards=false
• 关键词: EAGER; Nugget; Based; Information; Retrieval

Evaluating information retrieval systems, such as search engines, is critical to their effective development. Current performance evaluation methodologies are generally variants of the Cranfield paradigm, which relies on effectively complete, and thus prohibitively expensive, relevance judgment sets: tens to hundreds of thousands of documents must be judged by human assessors for relevance with respect to dozens to hundreds of user queries, at great cost both in time and expense. This exploratory project investigates a new alternative to information retrieval evaluation paradigm -- based on "nuggets". "Nuggets" are atomic units of relevant information, and one instantiation of these nuggets is simply the sentence or short passage that causes a judge to deem a document relevant at the time of document assessment. The hypothesis is that while it is likely impossible to find all relevant documents for a query with respect to web-scale and/or dynamic collections, it is much more tractable to find all or nearly all nuggets (i.e., relevant information), with which one can then perform effective and reusable evaluation, at scale and with ease. At evaluation time, relevance assessments are dynamically created for documents based on the quantity and quality of relevant information found in the documents retrieved. This new evaluation paradigm is inherently scalable and permits the use of all standard measures of retrieval performance, including those involving graded relevance judgments, novelty, diversity, and so on; it further permits new kinds of evaluations not heretofore possible.The project plan includes the development and release of nugget-based evaluation data sets for use by academia and industry. In fostering this effort, the project team has close ties with the US National Institute of Standards and Technology (NIST) and the Japanese National Institute of Informatics (through NTCIR), two of the premier organizations that develop and release information retrieval data sets. All research results and data sets developed as part of this project are available at the project website (http://www.ccs.neu.edu/home/jaa/IIS-1256172/). The project also provides educational and training experience for students and the development of curricular materials based on the project results.

评价信息检索系统，如搜索引擎，是其有效发展的关键。当前的性能评估方法通常是克兰菲尔德范例的变体，该范例依赖于有效地完整的、因此昂贵得令人望而却步的相关性判断集：必须由人类评估员判断数万到数十万个文档与数十到数百个用户查询的相关性，这在时间和费用上都是巨大的成本。本研究探讨一种新的资讯撷取评估模式--以“金块”为基础。 “金块”是相关信息的原子单位，这些金块的一个实例是简单的句子或简短的段落，导致法官在评估文件时认为文件相关。假设是，虽然可能不可能找到关于网络规模和/或动态集合的查询的所有相关文档，但是找到所有或几乎所有的金块（即，相关信息），然后可以大规模轻松地进行有效和可重复使用的评估。在评估时，基于在检索到的文档中发现的相关信息的数量和质量，动态地为文档创建相关性评估。这种新的评估范式本质上是可扩展的，允许使用所有检索性能的标准衡量标准，包括涉及分级相关性判断、新奇、多样性等的衡量标准;它还允许迄今为止不可能的新型评估。该项目计划包括开发和发布基于金块的评估数据集，供学术界和工业界使用。在促进这一努力的过程中，该项目小组与美国国家标准与技术研究所（NIST）和日本国家信息学研究所（通过NTCIR）有着密切的联系，这两个机构是开发和发布信息检索数据集的主要组织。作为本项目一部分开发的所有研究结果和数据集均可在项目网站（http：//www.ccs.neu.edu/home/jaa/IIS-1256172/）上获取。该项目还为学生提供教育和培训经验，并根据项目结果编写教材。