Improving Panel Decision Making: Understanding Methods for Aggregating Reviewer Opinions

该研究项目将开发数学和统计建模以及机器学习方法，以了解同行评审小组的决策过程。许多高风险的决定，如赠款资金或候选人雇用涉及同行评审小组。在这些小组中，合格的个人审查员通过预定义的程序对赠款提案或职位候选人提出意见。尽管小组成员对单个申请可能有多种意见，但小组一级的同行评审结果往往是一个数字，例如评审员分数的平均值。本项目将系统地研究将个人意见汇总为小组一级决定的现有方法。它还将开发一套工具，向利益攸关方传达小组决策信息。将开发的工具将使小组决策过程更加透明。利益相关者将能够识别出共识较少、不确定性较多的应用程序，这可能会对人类判断中的偏见产生直接影响。该项目将利用从各种同行评审过程中收集的真实的数据完善新工具，并开发新的方法来模拟和表示小组共识的不确定性和稳健性，无论是围绕一个主要的应用排名，还是作为根据潜在意见组提供不同排名的不同意见。将开发一个名为改进小组共识工具或ImPaCT的决策支持系统。ImPaCT将通过一组可视化工具呈现来自小组评审的信息和意见，这些工具可以显示应用程序之间的得分和相对顺序，指示哪些应用程序具有可比性，并且如果需要，为评审人员分配重要性权重以确定是否会改变结果。ImPaCT可作为一种分析工具，用于为资助机构和项目官员等利益攸关方了解和总结审评员的科学价值评估，或作为一种互动工具，协助专家组成员做出专家组层面的科学价值决策。ImPaCT将提供任务概要，标记需要额外注意的问题（关系、缺乏共识或鲁棒性），并提供当前子任务的相关信息（例如，正在讨论中）。通过在真实的时间内提供评论者评估的概要，这个工具将有助于检查什么可能是最响亮的声音的不适当的突出以及锚定，选择性记忆的不适当影响，和其他认知偏见的个人判断的复杂信息的集合下的不确定性。这一奖项反映了NSF的法定使命，并已被认为是值得支持，通过评估使用基金会的学术价值和更广泛的影响评审标准。

This research project will develop mathematical and statistical modeling as well as machine learning methods for understanding the decision-making process of peer review panels. Many high-stake decisions such as grant funding or candidate hiring involve peer review panels. In these panels, qualified individual reviewers provide their opinion on grant proposals or job candidates via a predefined process. Despite a potential plurality of opinions among panel members for a single application, the panel-level outcome of peer review often is a single number, such as the average of reviewers' scores. This project will systematically study existing methods for aggregating individual opinions into panel-level decisions. It also will develop a set of tools to communicate panel decision-making information to stakeholders. The tools to be developed will make the panel decision-making process more transparent. Stakeholders will be able to identify applications where there is less consensus and more uncertainty, potentially having direct impact on biases in human judgements. The project will refine the new tools with real data collected from various peer review processes.This research project will develop new methodologies to model and represent uncertainty and robustness in panel consensus, whether around one dominant ranking of applications or as divergent opinions that provide different rankings depending on the latent opinion group. A decision support system called Improving Panel Consensus Tool or ImPaCT will be developed. ImPaCT will present the information and opinions from panel review via a set of visualization tools that can display score and relative order between applications, indicate which applications are comparable, and, if desired, assign importance weights to reviewers to determine if that changes the outcome. ImPaCT could be used as an analysis tool for understanding and summarizing reviewers' scientific merit assessments for stakeholders such as funding agencies and program officials or as an interactive tool for assisting panel members with making panel-level scientific merit decisions. ImPaCT will present a synopsis of the task, flag issues needing extra attention (ties, lack of consensus, or robustness) and offer the relevant information for the current sub-task (e.g., submission under discussion). By providing a synopsis of reviewers' assessments in real time, this tool will help keep in check what could otherwise be the undue salience of the loudest voices as well as the undue influence of anchoring, selective memory, and other cognitive biases on individual judgment in the aggregation of complex information under uncertainty.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.