权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

Developing Methodology for Commensuration Bias Detection in Grant Application Peer Review

开发拨款申请同行评审中的补偿偏差检测方法

基本信息

批准号：
1759825
负责人：
Elena Erosheva
金额：
$ 26万
依托单位：
University of Washington
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-04-01 至 2021-03-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1759825&HistoricalAwards=false
关键词：
Developing Methodology Commensuration Bias Detection

项目摘要

Through its six federal grant agencies, the United States invests billions of dollars annually to promote science, technology, and engineering research at colleges and universities. The long-term goal of this public investment of money and trust is to improve our nation's health, economy, and social policies. In order to determine which researchers and projects will receive funding, grant agencies employ a process of peer review in which expert researchers evaluate the merits of submitted proposals. This longstanding social technology -- of relying on expert evaluation to inform determinations of merit -- empowers grant agencies to make funding decisions based on a fuller understanding of the social and scientific excellence of each project. As such, it is critical for grant agencies to employ rigorous and fair peer review processes in order to recruit, retain, and fund the best minds. This project builds on a growing scientific literature that studies how grant peer review works with an eye towards identifying ways of improving its effectiveness. More specifically, grant proposal review procedures commonly require reviewers to score applications along multiple dimensions -- for example, a proposal's approach, innovation, versus significance -- as an intermediate step in determining the proposal's overall score. When procedures are left unspecified for how reviewers should combine individual scores (along multiple dimensions) into overall scores, evaluators might arrive at overall scores in ways that subtly advantage and disadvantage grant proposals submitted by applicants from different social groups. Any such difference is what we call commensuration bias. This research identifies and evaluates approaches for measuring commensuration bias by analyzing peer review data from applications submitted to an ongoing intramural collaborative biomedical research program that utilized the independent peer review services of the American Institute of Biological Sciences. This project aims to offer concrete, efficient policies that ensure fair review for any grant agency that requires scoring of applications along multiple criteria, including the National Institutes of Health, which is the world's largest public funder of biomedical research in the world.There is currently no established methodology for detecting commensuration bias. The availability of criteria and overall scores from individual reviewers makes it possible to examine how individual reviewers evaluate multiple criteria simultaneously and how they unconsciously combine criteria scores to arrive at overall scores. In addition to hierarchical linear models that assume overall application score is an additive function of criteria scores, this study uses Bayesian regression trees to model non-linear decision processes and multivariate analyses to model the distribution of criteria scores. This range of statistical approaches allows this study to avoid making strong assumptions about the nature of commensuration and provides tools needed to inform two main types of potential policy recommendations: (a) those that focus on the restructuring or modification of programmatic review procedures as a whole and (b) those that focus on the monitoring of unusual peer review scores.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

通过六个联邦拨款机构，美国每年投资数十亿美元，以促进学院和大学的科学、技术和工程研究。这种资金和信任的公共投资的长期目标是改善我们国家的健康，经济和社会政策。为了确定哪些研究人员和项目将获得资助，资助机构采用了同行评审的程序，由专家研究人员评估提交的提案的优点。这种长期存在的社会技术----依靠专家评估来确定价值----使赠款机构能够在更充分地了解每个项目的社会和科学优点的基础上作出资助决定。因此，资助机构必须采用严格和公平的同行评审程序，以招募、留住和资助最优秀的人才。该项目建立在越来越多的科学文献的基础上，这些文献研究了赠款同行评审如何工作，以期确定提高其有效性的方法。更具体地说，拨款申请审查程序通常要求审查人员从沿着多个方面对申请进行评分--例如，申请的方法、创新性和重要性--作为确定申请总体得分的中间步骤。当评审员不知道如何将联合收割机的个人评分（沿着多个维度）结合成总分时，评审员可能会以微妙的方式得出总分，从而对来自不同社会群体的申请人提交的资助申请产生有利或不利的影响。任何这样的差异就是我们所说的解释偏差。本研究确定和评估的方法来衡量通过分析同行评审数据提交给一个正在进行的校内合作生物医学研究计划，利用独立的同行评审服务的美国生物科学研究所的应用程序的偏差。该项目旨在提供具体、有效的政策，以确保对任何需要根据沿着多个标准对申请进行评分的资助机构进行公平审查，其中包括美国国立卫生研究院（National Institutes of Health），该研究院是世界上最大的生物医学研究公共资助机构。个体评审员的标准和总体评分的可用性使得有可能检查个体评审员如何同时评估多个标准以及他们如何无意识地将联合收割机标准评分组合以获得总体评分。除了分层线性模型，假设整体应用程序的分数是一个加性函数的标准分数，本研究使用贝叶斯回归树模型的非线性决策过程和多变量分析模型的标准分数的分布。这一系列统计方法使本研究避免了对通货膨胀的性质作出强有力的假设，并提供了必要的工具，为两种主要类型的潜在政策建议提供信息：（a）侧重于整个方案审查程序的结构调整或修改的方案，以及（B）该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准。