Evaluation of High-Recall Human-in-the-Loop Information Retrieval Technologies

高查全率信息检索（HRIR）的目标是识别与信息需求相关的基本所有信息，其中缺失或不及时的结果可能会产生严重的法律的、政策、健康、社会、安全、国防或财务影响。 为了在实践中获得认可，HRIR技术必须更有效，并且必须根据管理应用领域的法律的、法令、监管、道德或专业标准，证明其比当前实践更有效。 申请人是这一领域的领先专家，首先认识到机器学习技术从垃圾邮件过滤到电子发现的可转移性，然后为美国和欧洲的民事诉讼和监管程序接受这些技术奠定了科学基础。 拟议中的研究包括实验，旨在提高和扩展HRIR的结果取得了日期，在应用领域，它是至关重要的，以满足信息需求时，它出现，并在一个持续的基础上，找到基本上所有相关的信息。 为了实现这些目标，将采取四种一般的调查途径：（1）使用实时数据和真实的信息需求来衡量HRIR技术在实践中的有效性;（2）对更细微的用户-系统交互进行建模，并采用更高保真的评估措施;（3）解决大规模数据集和高容量数据流带来的技术和后勤挑战;（4）发现和评估HRIR技术在新应用领域的有效性。 这些领域包括，但不限于，在政府档案的管理中区分公共和非公共记录;循证医学中荟萃分析的系统性审查;在会计重述中将违规行为和故意错报与无意错误分开;对未决的合并、收购和融资交易进行“尽职调查”;以及涉及大量数据集的监控和合规活动。拟议的研究有望建立HRIR技术在这些领域以及其他领域的有效性。

The objective of high-recall information retrieval (HRIR) is to identify substantially all information relevant to an information need, where the consequences of missing or untimely results may have serious legal, policy, health, social, safety, defence, or financial implications. To find acceptance in practice, HRIR technologies must be more effectiveand must be shown to be more effectivethan current practice, according to the legal, statutory, regulatory, ethical, or professional standards governing the application domain. The applicant is the leading expert in this field, having first recognized the transferability of machine-learning technologies from spam filtering to electronic discovery, and then having established the scientific foundation for their acceptance in civil litigation and regulatory proceedings, both in the United States and Europe. The proposed research includes experiments seeking to enhance and extend the HRIR results achieved to date, within application domains where it is critical to satisfy an information need when it arises, and on an ongoing basis, by finding substantially all relevant information. In furtherance of these goals, four general avenues of investigation will be pursued: (1) measuring the effectiveness of HRIR technologies in practice using live data and real information needs; (2) modeling more nuanced user-system interactions and employing higher-fidelity evaluation measures; (3) addressing the technological and logistical challenges presented by massive datasets and high-volume data streams; and (4) discovering and evaluating the effectiveness of HRIR technologies in new applications domains. Such domains include, but are not limited to, distinguishing between public and non-public records in the curation of government archives; systematic review for meta-analysis in evidence-based medicine; separating irregularities and intentional misstatements from unintentional errors in accounting restatements; performing “due diligence” in connection with pending mergers, acquisitions, and financing transactions; and surveillance and compliance activities involving massive datasets. The proposed research promises to establish the effectiveness of HRIR technology over current practice in those domains, as well as others.