HARNESSING MACHINE LEARNING ALGORITHMS TO STUDY SCIENTIFIC GRANT PEER REVIEW

凭借300亿美元的年度预算，美国国立卫生研究院(NIH)在为促进人类健康和疾病治疗的研究提供资金方面处于世界领先地位。像大多数资助机构一样，NIH使用同行评审来评估拨款申请的价值。在参加研究小组会议之前，大约有三名来自特定审查小组(称为“研究小组”)的评审员分配初步影响分数，并撰写评论以评估每个申请，所有成员都对最终优先分数做出贡献。尽管NIH的审查程序被认为是世界上最好的程序之一，但报告和自我研究表明，种族/少数民族和妇女首次获奖率和续签申请分别较低，NIH最大的资助机制R01拨款。这是有问题的，因为R01对职业发展至关重要，而由种族/少数民族和妇女进行的研究与技术创新有关，并以解决昂贵的教育、经济和健康差距而闻名。作为努力使科学和医疗人员多样化的领导者，NIH呼吁进行研究，以测试在其同行审查过程中存在偏见的可能性。这一呼吁揭示了对同行评议的有效性进行研究的广泛需要，这种研究在所有科学和技术领域都得到了应用，并需要更多的科学家参与这种研究。如果与拟议科学的质量无关的因素对拨款审查的结果产生负面影响，这与资助机构选择最佳科学的目标背道而驰，阻碍了昂贵的下游联邦政府扩大对科学的参与的努力，并削弱了美国科学企业的竞争力。我们的小组首次表明，当与对分数和获奖率的传统分析相结合时，对NIH同行评审员对R01申请的叙事批评的语言分析可以显示出评审员决策中潜在的基于刻板印象的偏见的证据。尽管这种偏见通常是无意的，影响了评审者的判断，无论他们自己的性别或种族，但它可能会导致评审者以不同的方式执行评估标准。例如，对照实验表明，认为少数族裔和女性在科学等领域缺乏内在能力的文化刻板印象可能会导致评审者下意识地要求更多证据来证实他们的能力。在过去的十年里，机器学习技术已经将数据挖掘、文本挖掘和视频挖掘转化为最先进的分析技术，如果应用于科学同行审查，可能会给该领域带来革命性的变化。长短期记忆(LSTMS)神经网络的功能就像人脑一样识别数据中的复杂模式，尤其是推动了计算机科学在社会和心理现象研究中的应用。使用大量人口统计学上不同的NIH R01申请评论、分数和构建的研究部分讨论视频，该项目正在开发分析工具，使用LSTM在拨款申请的书面和口头讨论中捕捉基于刻板印象的偏见的证据。由此产生的技术是开放获取的，并可用于科学资助机构的应用。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Armed with a $30 billion annual budget, the U.S. National Institutes of Health (NIH) leads the world in funding research to advance human health and treatments for disease. Like most funding agencies, NIH uses peer review to evaluate the merit of grant applications. Approximately three reviewers from a given review group (called a "study section") assign preliminary impact scores and write critiques to evaluate each application before attending study section meetings where all members contribute to a final priority score. Although NIH's review process is considered one of the best in the world, reports and self-studies show that racial/ethnic minorities and women have lower award rates for first time, and renewal applications, respectively, for NIH's largest funding mechanism, the R01 grant. This is problematic because R01s are critical for career advancement, and research conducted by racial/ethnic minorities and women is linked to technological innovation and is known to address costly education, economic, and health disparities. As a leader in efforts to diversify the science and medical workforce, NIH has called for studies to test for the possibility that bias may operate in its peer review process. This call brings to light the broad need for research on the effectiveness of peer review, which is used across all science and technology fields, and for more scientists to engage in such research. If factors unrelated to the quality of the proposed science negatively impact the outcome of a grant review, it runs counter to funding agencies' goals to select the best science, blocks expensive downstream federal efforts to broaden participation in science, and undermines the competitiveness of the U.S. scientific enterprise.Our group was the first to show that, when combined with traditional analyses of scores and award rates, linguistic analysis of NIH peer reviewers' narrative critiques of R01 applications can show evidence of potential stereotype-based bias in reviewers' decision making. Although such bias is generally unintentional and impacts reviewers' judgment regardless of their own sex or race, it can lead reviewers to differentially enforce evaluation criteria. Controlled experiments show, for instance, that cultural stereotypes that racial/ethnic minorities and women lack intrinsic ability for fields like science, can lead reviewers to unconsciously require more proof to confirm their competence. Over the past decade machine learning technologies have made data-, text-, and video-mining into state-of-the-art analytic techniques, which, if applied to scientific peer review, could revolutionize the field. Long Short Term Memory (LSTMs) neural networks -- algorithms that function like the human brain to identify complex patterns in data -- in particular, have catapulted the application of computer science to the study of social and psychological phenomena. Using a large, demographically diverse set of NIH R01 application critiques, scores, and video of constructed study section discussions, this project is producing analytical tools that use LSTMs to capture evidence of stereotype-based bias in both written and oral discussion of grant applications. Resulting technologies are open-access, and available for applied use across scientific funding agencies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.