Exploring the Science of Scientific Review

描述(由申请人提供)：所有人--无论多么聪明、平等主义或善意--在做出决定和评判他人的方式上都容易受到认知偏见的影响。尽管这些偏见可能会在无意中与一个人的有意识的意图、个人信念和客观数据背道而驰，但它们也可能在无意中使社会不平等永久化。R01资助结果中种族和性别的不明差异引起了人们对NIH同行审查中的偏见的担忧。这一变革性的R01将通过以下三个具体目标检查R01同行评议中是否以及如何可能发生隐性(即无意的)偏见：具体目标1.确定调查员特征对R01同行评审员选择的词语和描述符的影响程度，以及文本与分配分数的关系。我们将验证正面和负面拨款评估词类，分析全国R01审查样本的文本，并比较不同研究人员特征的拨款审查文本。我们假设，即使在申请分数和资助结果相似的情况下，词汇和描述符的类别也会有所不同，从而暗示着申请者种族和性别的不同评估标准。具体目标#2.确定调查员种族、性别或机构是否会对相同提案的审查产生因果影响。我们 我们将进行一项随机对照研究，在该研究中，我们操纵拨款首席调查员(PI)的特征，以评估他们对拨款审查结果的影响。我们将请求实际资助的R01的捐款，并在每一笔赠款中操纵PI的性别、种族或家庭机构。然后，我们将邀请适当学科的评审员对提案进行评审，并对书面评审和分数进行分析。我们假设，调查员变量将显著影响分数和审查文本，从而将更高的拨款归因于 状态组(男性、白人、著名机构)将获得更好的分数，文本将隐含地支持不同的卓越标准。具体目标#3.研究研究小组成员之间的互动模式如何提高对讨论主题和相关资助申请者的接受度和抵抗力。在构建的研究部分的录音和录像中，我们将调查R01提案讨论中的实时社会互动过程。我们将使用会话分析来检查初始排名的交付及其理由， 主题发展，以及协商最终排名的过程。这项研究是创新的，因为它首次检查了潜在偏见的复杂性 美国国立卫生研究院同行评审。这项研究的潜在影响有三个方面：1)发现某些形式的认知偏见在R01同行评议中是否有影响；2)确定定量文本分析是否是内隐偏见的有用测量；3)描述和标记实时格兰特评审员的互动模式。综上所述，我们的研究结果可以为整个NIH同行审查的转变奠定基础。

DESCRIPTION (provided by applicant): All humans-no matter how intelligent, egalitarian, or well-intentioned-are susceptible to cognitive biases in the way they make decisions and judge others. Although these biases can operate unintentionally in opposition to one's conscious intentions, personal beliefs, and objective data, they may unwittingly perpetuate social inequalities. Unexplained disparities in R01 funding outcomes by race and gender have raised concern about bias in NIH peer review. This Transformative R01 will examine if and how implicit (i.e., unintentional) bias might occur in R01 peer review through the following three Specific Aims: Specific Aim #1. Identify the extent to which investigator characteristics influence the words and descriptors chosen by R01 peer-reviewers and how text relates to assigned scores. We will validate positive and negative grant evaluation word categories, analyze the text of a national sample of R01 reviews, and compare the grant review text for different investigator characteristics. We hypothesize that categories of words and descriptors will differ in ways that suggest implicitly different evaluation standards by applicant race and gender, even when application scores and funding outcomes are similar. Specific Aim #2. Determine whether investigator race, gender, or institution causally influences the review of identical proposals. We will conduct a randomized, controlled study in which we manipulate characteristics of a grant principal investigator (PI) to assess their influence on grant review outcomes. We will request donations of actual funded R01s and, within each grant, manipulate the PI's gender, race, or home institution. We will then invite reviewers in the appropriate discipline to review the proposals, and we will analyze written reviews and scores. We hypothesize that investigator variables will significantly influence scores and review text such that grants attributed to higher status groups (male, White, prestigious institution) will obtain better scores and text will suppor implicitly different standards of excellence. Specific Aim #3. Examine how interactional patterns among study section members promote receptivity and resistance to discussion topics and associated grant applicants. In audio- and videotapes of constructed study sections, we will investigate the real-time social interactional processes in the discussions of R01 proposals. We will employ conversation analysis to examine the delivery of initial rankings and their rationales, topic development, and the processes through which final rankings are negotiated. This research is innovative because it examines for the first time the complexities of potential bias in NIH peer review. The potential impact is threefold; this research will 1) discover whether certain forms of cognitive bias are or are not consequential in R01 peer review, 2) determine whether quantitative text analysis is a useful measure of implicit bias, and 3) describe and label real-tim grant reviewer interactional patterns. Taken together, the results of our research could set the stage for transformation in peer review throughout NIH.