III: Small: Collaborative Research: Scrutable and Explainable Information Retrieval with Model Intrinsic and Agnostic Approaches

信息检索系统是人们获取信息的重要手段。例如，智能搜索引擎被广泛应用于诸如网页搜索、产品搜索和工作搜索的基于Web的服务中。最近，复杂的数据和复杂的黑盒模型使得现代IR系统对用户不那么透明。然而，随着越来越多的人依赖于IR系统来指导他们的日常生活和决策制定，对于技术社区和普通公众来说，对可解释的搜索结果的需求不断增长，以便他们理解为什么提供某些搜索结果。与此同时，政府机构要求IR系统不仅要提供高质量的结果，而且要提供合理的理由，以提高系统的可信度。该项目的重点是开发算法和框架，以提高现代IR系统的可理解性，可解释性和透明度。它将激发大规模的学术-产业合作，通过促进可靠和可解释的信息访问服务的开发，使数十亿用户受益。本计画将开发可解释与可理解的资讯检索系统的一般性与可重复使用的架构。第一个方向是针对模型内在解释的新检索模型。这包括为检索操作开发透明的推理过程和决策边界，支持用户反馈的结果探索的可理解功能，以及可跟踪的信息流，以区分模型输入的贡献。第二个方向是建立模型不可知论解释的分析和模拟框架。这包括事后解释系统与外部知识，和一个模拟框架，在黑盒检索模型与可解释的输出。除了模型内在和模型不可知的方法，这个项目还将研究众包任务和系统的度量，以比较内在和不可知的解释的有效性。研究成果将包括多个公共基准数据集和可解释IR的评估平台，这将有助于研究界进行可持续和可重复的未来研究。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

Information Retrieval (IR) systems are important for people for information access. For example, intelligent search engines are widely used in Web-based services such as web search, product search, and job search. Recently, sophisticated data and complicated black-box models have made modern IR systems less transparent to users. However, as more and more people rely on IR systems to guide their daily life and decision making, there has been growing needs of explainable search results, both for technical communities and the general public, so that they understand why certain search results are provided. Meanwhile, governmental agencies are demanding IR systems to provide not only high-quality results, but also reasonable justifications, so as to enhance the trustworthiness of the systems. This project focuses on developing algorithms and frameworks to improve the scrutability, explainability, and transparency of modern IR systems. It will inspire large-scale academic-industry collaboration, which benefits billions of users by facilitating the development of reliable and explainable information access services. This project will develop general and reusable frameworks for scrutable and explainable IR. Research in this project will be performed on two directions. The first direction aims at new retrieval models for model-intrinsic explanation. This includes developing transparent inference process and decision boundaries for retrieval actions, scrutable functions that support result exploration with user feedback, and traceable information flow to distinguish the contribution of model inputs. The second direction aims at building analytical and simulative framework for model-agnostic explanation. This includes post-hoc explanation systems with external knowledge, and a simulation framework over black-box retrieval models with explainable outputs. Besides model-intrinsic and model-agnostic approaches, this project will also investigate crowd-sourcing tasks and systematic metrics to compare the effectiveness of intrinsic and agnostic explanations. The research outcomes will include multiple public benchmark datasets and evaluation platforms for explainable IR, which will contribute to the research community for sustainable and reproducible future studies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.