Exploring the Science of Scientific Review

描述（由申请人提供）：所有的人-无论多么聪明，平等主义者，或善意-在他们做出决定和判断他人的方式中容易受到认知偏见的影响。虽然这些偏见可能会无意中与一个人的有意识的意图、个人信仰和客观数据相对立，但它们可能会无意中使社会不平等永久化。R 01资助结果在种族和性别方面存在无法解释的差异，这引起了人们对NIH同行评审中偏见的担忧。本转型R 01将检查是否以及如何隐含（即，在R 01同行评审中，通过以下三个具体目标可能会出现（无意的）偏见：具体目标#1。确定研究者特征对R 01同行评审员选择的词语和描述符的影响程度，以及文本与指定分数的关系。我们将验证积极和消极的资助评估词类别，分析R 01评论的国家样本的文本，并比较不同研究者特征的资助评论文本。我们假设，即使申请分数和资助结果相似，词语和描述符的类别也会有所不同，从而暗示申请人种族和性别的评估标准会有所不同。 具体目标#2。确定研究者种族、性别或机构是否会对相同提案的审查产生因果影响。我们 我们将进行一项随机对照研究，在这项研究中，我们操纵了一名研究基金主要研究者（PI）的特征，以评估他们对研究基金审查结果的影响。我们将要求捐赠实际资助的R 01，并在每个赠款中，操纵PI的性别，种族或家庭机构。然后，我们将邀请相应学科的评审员对提案进行评审，我们将分析书面评论和分数。我们假设研究者变量将显著影响分数和审查文本，这样赠款归因于更高的补助金 地位群体（男性、白色、著名机构）将获得更好的分数，文本将隐含地支持不同的优秀标准。 具体目标#3检查研究组成员之间的干扰模式如何促进对讨论主题和相关补助金申请人的接受和抵制。在构建的学习部分的音频和视频中，我们将调查R 01提案讨论中的实时社交互动过程。我们将采用会话分析来检查初始排名的交付及其理由， 主题开发，以及最终排名谈判的过程。 这项研究是创新的，因为它第一次研究了潜在偏见的复杂性， NIH同行评审。潜在的影响是三方面的;本研究将1）发现某些形式的认知偏见是否在R 01同行评审中是或不是重要的，2）确定定量文本分析是否是一个有用的衡量隐式偏见的方法，3）描述和标记实时资助评审员的干扰模式。总的来说，我们的研究结果可以为整个NIH的同行评议改革奠定基础。

DESCRIPTION (provided by applicant): All humans-no matter how intelligent, egalitarian, or well-intentioned-are susceptible to cognitive biases in the way they make decisions and judge others. Although these biases can operate unintentionally in opposition to one's conscious intentions, personal beliefs, and objective data, they may unwittingly perpetuate social inequalities. Unexplained disparities in R01 funding outcomes by race and gender have raised concern about bias in NIH peer review. This Transformative R01 will examine if and how implicit (i.e., unintentional) bias might occur in R01 peer review through the following three Specific Aims: Specific Aim #1. Identify the extent to which investigator characteristics influence the words and descriptors chosen by R01 peer-reviewers and how text relates to assigned scores. We will validate positive and negative grant evaluation word categories, analyze the text of a national sample of R01 reviews, and compare the grant review text for different investigator characteristics. We hypothesize that categories of words and descriptors will differ in ways that suggest implicitly different evaluation standards by applicant race and gender, even when application scores and funding outcomes are similar. Specific Aim #2. Determine whether investigator race, gender, or institution causally influences the review of identical proposals. We will conduct a randomized, controlled study in which we manipulate characteristics of a grant principal investigator (PI) to assess their influence on grant review outcomes. We will request donations of actual funded R01s and, within each grant, manipulate the PI's gender, race, or home institution. We will then invite reviewers in the appropriate discipline to review the proposals, and we will analyze written reviews and scores. We hypothesize that investigator variables will significantly influence scores and review text such that grants attributed to higher status groups (male, White, prestigious institution) will obtain better scores and text will suppor implicitly different standards of excellence. Specific Aim #3. Examine how interactional patterns among study section members promote receptivity and resistance to discussion topics and associated grant applicants. In audio- and videotapes of constructed study sections, we will investigate the real-time social interactional processes in the discussions of R01 proposals. We will employ conversation analysis to examine the delivery of initial rankings and their rationales, topic development, and the processes through which final rankings are negotiated. This research is innovative because it examines for the first time the complexities of potential bias in NIH peer review. The potential impact is threefold; this research will 1) discover whether certain forms of cognitive bias are or are not consequential in R01 peer review, 2) determine whether quantitative text analysis is a useful measure of implicit bias, and 3) describe and label real-tim grant reviewer interactional patterns. Taken together, the results of our research could set the stage for transformation in peer review throughout NIH.