Evaluation of High-Recall Human-in-the-Loop Information Retrieval Technologies

高查全率信息检索(HRIR)的目标是识别基本上所有与信息需求相关的信息，在这些信息中，丢失或不及时的结果可能会产生严重的法律、政策、健康、社会、安全、国防或财务影响。为了在实践中获得认可，根据管理应用领域的法律、法规、法规、道德或专业标准，HRIR技术必须更有效，并且必须证明比目前的做法更有效。申请人是这一领域的领先专家，他首先认识到机器学习技术从垃圾邮件过滤到电子发现的可转移性，然后为在美国和欧洲的民事诉讼和监管程序中接受这些技术奠定了科学基础。 拟议的研究包括旨在加强和扩大迄今已取得的HRIR成果的实验，在应用领域内，在出现信息需求时满足信息需求是至关重要的，并在持续的基础上，通过查找几乎所有相关信息。为实现这些目标，将采用四种一般调查途径：(1)利用现场数据和真实信息需求衡量高分辨率信息检索技术在实践中的有效性；(2)建立更细微差别的用户-系统互动模型，并采用更高保真的评估措施；(3)解决海量数据集和大量数据流带来的技术和后勤挑战；(4)发现和评估高分辨率信息检索技术在新的应用领域的有效性。这些领域包括但不限于，在政府档案管理中区分公共和非公共记录；循证医学中的荟萃分析的系统审查；将会计重述中的违规和故意错误陈述与无意错误分开；对未决的合并、收购和融资交易进行“尽职调查”；以及涉及海量数据集的监督和合规活动。拟议的研究承诺确定HRIR技术在这些领域和其他领域的当前实践中的有效性。

The objective of high-recall information retrieval (HRIR) is to identify substantially all information relevant to an information need, where the consequences of missing or untimely results may have serious legal, policy, health, social, safety, defence, or financial implications. To find acceptance in practice, HRIR technologies must be more effectiveand must be shown to be more effectivethan current practice, according to the legal, statutory, regulatory, ethical, or professional standards governing the application domain. The applicant is the leading expert in this field, having first recognized the transferability of machine-learning technologies from spam filtering to electronic discovery, and then having established the scientific foundation for their acceptance in civil litigation and regulatory proceedings, both in the United States and Europe. The proposed research includes experiments seeking to enhance and extend the HRIR results achieved to date, within application domains where it is critical to satisfy an information need when it arises, and on an ongoing basis, by finding substantially all relevant information. In furtherance of these goals, four general avenues of investigation will be pursued: (1) measuring the effectiveness of HRIR technologies in practice using live data and real information needs; (2) modeling more nuanced user-system interactions and employing higher-fidelity evaluation measures; (3) addressing the technological and logistical challenges presented by massive datasets and high-volume data streams; and (4) discovering and evaluating the effectiveness of HRIR technologies in new applications domains. Such domains include, but are not limited to, distinguishing between public and non-public records in the curation of government archives; systematic review for meta-analysis in evidence-based medicine; separating irregularities and intentional misstatements from unintentional errors in accounting restatements; performing “due diligence” in connection with pending mergers, acquisitions, and financing transactions; and surveillance and compliance activities involving massive datasets. The proposed research promises to establish the effectiveness of HRIR technology over current practice in those domains, as well as others.