RI: Small: Human Validation in Batch Reinforcement Learning

在很多情况下，尝试一项新决策可能代价高昂，但记录过去的决策和结果可能有助于为新决策提供信息。例如，健康记录跟踪临床决策和结果；在线课程可以跟踪不同的教学和参与策略以及最终表现；工厂可以跟踪不同的工艺选择和输出质量。过去日志中的信息可以防止我们犯同样的错误并改善结果。然而，从这些记录的数据中学习并不容易：并非所有可能的决定都可能尝试过，也并非所有相关信息都将被记录下来：例如，健康记录可能准确地记录了患者接受的实验室检查，但缺乏有关其家庭和工作环境的潜在相关信息。这些挑战使得系统很难推断遵循与当前实践不同的决策策略的影响。目前的方法主要有两种：一种是统计方法，它有很强的理论基础，但需要很多假设；以及那些基于人类专业知识的方法，这些方法可能很强大，但也可能出错。这项工作结合了统计和基于人的验证方法的优势，以帮助从记录的数据中确定有希望的决策策略。具体而言，该项目侧重于为两项主要任务整合人力和统计投入。第一个任务是将原始输入（测量历史）转换为人类可理解的表示，其中使用统计方法来提出用于定义或总结策略的表示，并且使用人工输入来确保表示是直观的，或者至少是可理解的。第二项任务是在做出不同决策时估计结果的差异。在这里，使用统计方法来形成初始估计，并确定哪些数据对该估计影响最大，并且使用人工输入来确定估计是否可靠，因为它特别依赖于这些数据。这两个构建块允许我们总结数据和策略，以及估计结果，然后用于评估给定策略和提出新策略。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

There exist many settings in which trying out a new decision might be costly, but logs of past decisions and outcomes might help to inform this new decision. For example, health records track clinical decisions and outcomes; online courses may track different teaching and engagement strategies and final performance; factories may track different process choices and output quality. Information from past logs may prevent us from making the same mistakes and improve outcomes. However, learning from such logged data is not easy: not all possible decisions may have been tried, and not all relevant information will have been recorded: for example, a health record may accurately contain what lab tests a patient received but lack potentially relevant information about their home and work environment. These challenges make it hard for systems to reason about the effect of following a different decision-making strategy than current practice. Current approaches fall into two main types: statistical methods, which have strong theoretical foundations but require many assumptions; and those based on human expertise, which can be strong but also fallible. This work brings together the strengths of statistical and human-based approaches to validation to help identify promising decision-making strategies from logged data.Specifically, the project focuses on integrating human and statistical inputs for two major tasks. The first is the task of converting the raw inputs (histories of measurements) into human-understandable representations, where statistical methods are used to proposed representations that will be useful for defining or summarizing a policy and human input is used to ensure that the representation is intuitive, or at least understandable. The second is the task of estimating differences in outcomes if a different decision is made. Here, statistical methods are used to form the initial estimate as well as identify what data are most influential to that estimate, and human input is used to determine whether the estimate is reliable given that it relies particularly on those data. These two building blocks, which allow us to summarize the data and a policy, as well as estimate outcomes, are then used for both evaluating a given policy and proposing new policies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.