Developing Methodology for Commensuration Bias Detection in Grant Application Peer Review

通过其六个联邦拨款机构，美国每年投资数十亿美元来促进学院和大学的科学、技术和工程研究。这笔公共资金和信托投资的长期目标是改善我们国家的健康、经济和社会政策。为了确定哪些研究人员和项目将获得资助，赠款机构采用同行审查程序，由专家研究人员评估提交的提案的优点。这种长期存在的社会技术--依靠专家评估来确定功绩--使赠款机构能够根据对每个项目在社会和科学上的卓越表现的更充分了解来做出资助决定。因此，赠款机构采用严格和公平的同行审查程序，以招聘、留住和资助最优秀的人才是至关重要的。这个项目建立在越来越多的科学文献的基础上，这些文献研究了赠款同行审查是如何发挥作用的，并着眼于确定提高其有效性的方法。更具体地说，拨款提案审查程序通常要求审查者从多个方面对申请进行评分--例如，提案的方法、创新和重要性--作为确定提案总体得分的中间步骤。如果没有具体说明评价者应该如何将个人分数(沿着多个维度)合并为总体分数，评估者可能会以不同社会群体申请者提交的微妙的优势和劣势拨款提案的方式得出总体分数。任何这样的差异都是我们所说的通度偏差。这项研究通过分析提交给正在进行的利用美国生物科学研究所的独立同行评审服务的内部协作生物医学研究计划的申请的同行评审数据，确定和评估衡量通度偏差的方法。该项目旨在提供具体、有效的政策，确保对任何要求按照多种标准对申请进行评分的赠款机构进行公平审查，包括美国国立卫生研究院，它是世界上最大的生物医学研究公共资助者。目前还没有检测通缩偏差的既定方法。单个评审者提供的标准和总体分数使其能够检查单个评价者如何同时评估多个标准，以及他们如何不知不觉地组合标准分数以得出总体分数。除了假设总体申请分数是标准分数的加法函数的分层线性模型外，本研究还使用贝叶斯回归树来模拟非线性决策过程，并使用多元分析来模拟标准分数的分布。这一系列的统计方法使这项研究可以避免对通约的性质做出强有力的假设，并提供了为两种主要类型的潜在政策建议提供信息所需的工具：(A)那些侧重于整个方案审查程序的重组或修改的建议，以及(B)那些侧重于监测不寻常的同行审查分数的建议。这一奖项反映了国家科学基金会的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Through its six federal grant agencies, the United States invests billions of dollars annually to promote science, technology, and engineering research at colleges and universities. The long-term goal of this public investment of money and trust is to improve our nation's health, economy, and social policies. In order to determine which researchers and projects will receive funding, grant agencies employ a process of peer review in which expert researchers evaluate the merits of submitted proposals. This longstanding social technology -- of relying on expert evaluation to inform determinations of merit -- empowers grant agencies to make funding decisions based on a fuller understanding of the social and scientific excellence of each project. As such, it is critical for grant agencies to employ rigorous and fair peer review processes in order to recruit, retain, and fund the best minds. This project builds on a growing scientific literature that studies how grant peer review works with an eye towards identifying ways of improving its effectiveness. More specifically, grant proposal review procedures commonly require reviewers to score applications along multiple dimensions -- for example, a proposal's approach, innovation, versus significance -- as an intermediate step in determining the proposal's overall score. When procedures are left unspecified for how reviewers should combine individual scores (along multiple dimensions) into overall scores, evaluators might arrive at overall scores in ways that subtly advantage and disadvantage grant proposals submitted by applicants from different social groups. Any such difference is what we call commensuration bias. This research identifies and evaluates approaches for measuring commensuration bias by analyzing peer review data from applications submitted to an ongoing intramural collaborative biomedical research program that utilized the independent peer review services of the American Institute of Biological Sciences. This project aims to offer concrete, efficient policies that ensure fair review for any grant agency that requires scoring of applications along multiple criteria, including the National Institutes of Health, which is the world's largest public funder of biomedical research in the world.There is currently no established methodology for detecting commensuration bias. The availability of criteria and overall scores from individual reviewers makes it possible to examine how individual reviewers evaluate multiple criteria simultaneously and how they unconsciously combine criteria scores to arrive at overall scores. In addition to hierarchical linear models that assume overall application score is an additive function of criteria scores, this study uses Bayesian regression trees to model non-linear decision processes and multivariate analyses to model the distribution of criteria scores. This range of statistical approaches allows this study to avoid making strong assumptions about the nature of commensuration and provides tools needed to inform two main types of potential policy recommendations: (a) those that focus on the restructuring or modification of programmatic review procedures as a whole and (b) those that focus on the monitoring of unusual peer review scores.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.