Improving Panel Decision Making: Understanding Methods for Aggregating Reviewer Opinions

该研究项目将开发数学和统计建模以及机器学习方法，以了解同行评议小组的决策过程。许多高风险的决策，如拨款或候选人招聘，都涉及到同行评审小组。在这些小组中，合格的个人审稿人通过预定义的流程提供他们对拨款提案或职位候选人的意见。尽管小组成员对单个应用程序可能有多个意见，但小组级别的同行评审结果通常是单个数字，例如审稿人的平均分数。该项目将系统地研究将个人意见汇总为小组级决策的现有方法。它还将开发一套工具，将小组决策信息传达给利益相关者。待开发的工具将使小组决策过程更加透明。利益相关者将能够识别共识较少、不确定性较大的应用程序，这可能对人类判断中的偏见产生直接影响。该项目将使用从各种同行评审过程中收集的真实数据来改进新工具。该研究项目将开发新的方法来模拟和代表小组共识中的不确定性和稳健性，无论是围绕应用程序的一个主导排名还是根据潜在意见组提供不同排名的不同意见。将开发一个决策支持系统，称为改进小组共识工具或影响。ImPaCT将通过一组可视化工具呈现来自小组审查的信息和意见，这些工具可以显示应用程序之间的分数和相对顺序，指示哪些应用程序是可比较的，并且，如果需要，为审查者分配重要性权重，以确定这是否会改变结果。ImPaCT可以作为一种分析工具，用于理解和总结审稿人对利益相关者（如资助机构和项目官员）的科学价值评估，也可以作为一种互动工具，帮助小组成员做出小组级别的科学价值决策。ImPaCT将提供任务的概要，标记需要额外注意的问题（联系、缺乏共识或健壮性），并提供当前子任务的相关信息（例如，正在讨论的提交）。通过实时提供审稿人评估的概要，该工具将有助于检查在不确定的情况下，在复杂信息的聚合中，可能出现的最大声声音的不适当突出，以及锚定、选择性记忆和其他认知偏见对个人判断的不适当影响。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This research project will develop mathematical and statistical modeling as well as machine learning methods for understanding the decision-making process of peer review panels. Many high-stake decisions such as grant funding or candidate hiring involve peer review panels. In these panels, qualified individual reviewers provide their opinion on grant proposals or job candidates via a predefined process. Despite a potential plurality of opinions among panel members for a single application, the panel-level outcome of peer review often is a single number, such as the average of reviewers' scores. This project will systematically study existing methods for aggregating individual opinions into panel-level decisions. It also will develop a set of tools to communicate panel decision-making information to stakeholders. The tools to be developed will make the panel decision-making process more transparent. Stakeholders will be able to identify applications where there is less consensus and more uncertainty, potentially having direct impact on biases in human judgements. The project will refine the new tools with real data collected from various peer review processes.This research project will develop new methodologies to model and represent uncertainty and robustness in panel consensus, whether around one dominant ranking of applications or as divergent opinions that provide different rankings depending on the latent opinion group. A decision support system called Improving Panel Consensus Tool or ImPaCT will be developed. ImPaCT will present the information and opinions from panel review via a set of visualization tools that can display score and relative order between applications, indicate which applications are comparable, and, if desired, assign importance weights to reviewers to determine if that changes the outcome. ImPaCT could be used as an analysis tool for understanding and summarizing reviewers' scientific merit assessments for stakeholders such as funding agencies and program officials or as an interactive tool for assisting panel members with making panel-level scientific merit decisions. ImPaCT will present a synopsis of the task, flag issues needing extra attention (ties, lack of consensus, or robustness) and offer the relevant information for the current sub-task (e.g., submission under discussion). By providing a synopsis of reviewers' assessments in real time, this tool will help keep in check what could otherwise be the undue salience of the loudest voices as well as the undue influence of anchoring, selective memory, and other cognitive biases on individual judgment in the aggregation of complex information under uncertainty.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.