HARNESSING MACHINE LEARNING ALGORITHMS TO STUDY SCIENTIFIC GRANT PEER REVIEW

利用机器学习算法研究科学资助同行评审

• 批准号: 1760092
• 负责人: You-Geon Lee
• 金额: $ 49.93万
• 依托单位: University of Wisconsin-Madison
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-07-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1760092&HistoricalAwards=false
• 关键词: HARNESSING; MACHINE; LEARNING; ALGORITHMS; STUDY

Armed with a $30 billion annual budget, the U.S. National Institutes of Health (NIH) leads the world in funding research to advance human health and treatments for disease. Like most funding agencies, NIH uses peer review to evaluate the merit of grant applications. Approximately three reviewers from a given review group (called a "study section") assign preliminary impact scores and write critiques to evaluate each application before attending study section meetings where all members contribute to a final priority score. Although NIH's review process is considered one of the best in the world, reports and self-studies show that racial/ethnic minorities and women have lower award rates for first time, and renewal applications, respectively, for NIH's largest funding mechanism, the R01 grant. This is problematic because R01s are critical for career advancement, and research conducted by racial/ethnic minorities and women is linked to technological innovation and is known to address costly education, economic, and health disparities. As a leader in efforts to diversify the science and medical workforce, NIH has called for studies to test for the possibility that bias may operate in its peer review process. This call brings to light the broad need for research on the effectiveness of peer review, which is used across all science and technology fields, and for more scientists to engage in such research. If factors unrelated to the quality of the proposed science negatively impact the outcome of a grant review, it runs counter to funding agencies' goals to select the best science, blocks expensive downstream federal efforts to broaden participation in science, and undermines the competitiveness of the U.S. scientific enterprise.Our group was the first to show that, when combined with traditional analyses of scores and award rates, linguistic analysis of NIH peer reviewers' narrative critiques of R01 applications can show evidence of potential stereotype-based bias in reviewers' decision making. Although such bias is generally unintentional and impacts reviewers' judgment regardless of their own sex or race, it can lead reviewers to differentially enforce evaluation criteria. Controlled experiments show, for instance, that cultural stereotypes that racial/ethnic minorities and women lack intrinsic ability for fields like science, can lead reviewers to unconsciously require more proof to confirm their competence. Over the past decade machine learning technologies have made data-, text-, and video-mining into state-of-the-art analytic techniques, which, if applied to scientific peer review, could revolutionize the field. Long Short Term Memory (LSTMs) neural networks -- algorithms that function like the human brain to identify complex patterns in data -- in particular, have catapulted the application of computer science to the study of social and psychological phenomena. Using a large, demographically diverse set of NIH R01 application critiques, scores, and video of constructed study section discussions, this project is producing analytical tools that use LSTMs to capture evidence of stereotype-based bias in both written and oral discussion of grant applications. Resulting technologies are open-access, and available for applied use across scientific funding agencies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

美国国立卫生研究院（NIH）拥有300亿美元的年度预算，在资助促进人类健康和疾病治疗的研究方面处于世界领先地位。像大多数资助机构一样，NIH使用同行评审来评估拨款申请的价值。来自一个特定审查小组（称为“研究组”）的大约三名审查员在参加研究组会议之前分配初步影响分数并撰写评论以评估每个应用程序，所有成员都为最终优先分数做出贡献。虽然NIH的审查程序被认为是世界上最好的审查程序之一，但报告和自我研究表明，对于NIH最大的资助机制R 01赠款，种族/族裔少数群体和妇女的首次奖励率较低，而续签申请则分别较低。这是有问题的，因为R 01对职业发展至关重要，而少数民族和妇女进行的研究与技术创新有关，并且已知可以解决昂贵的教育，经济和健康差距。作为努力使科学和医学劳动力多样化的领导者，NIH呼吁进行研究，以测试在同行评审过程中存在偏见的可能性。这一呼吁揭示了广泛需要研究同行审查的有效性，这是在所有科学和技术领域使用，并为更多的科学家从事这种研究。如果与所提议的科学的质量无关的因素对资助审查的结果产生负面影响，这与资助机构选择最佳科学的目标背道而驰，阻碍了昂贵的下游联邦努力，以扩大科学的参与，并破坏了美国科学企业的竞争力。我们的小组是第一个表明，当结合传统的分数和奖励率分析时，对NIH同行评审员对R 01申请的叙述性批评的语言分析可以显示评审员决策中潜在的基于刻板印象的偏见的证据。虽然这种偏见通常是无意的，并影响评审员的判断，无论他们自己的性别或种族，它可以引导评审员有区别地执行评价标准。例如，对照实验表明，认为少数民族和女性缺乏科学等领域内在能力的文化刻板印象，可能会引导审查人员无意识地要求更多证据来确认他们的能力。在过去的十年中，机器学习技术已经将数据、文本和视频挖掘变成了最先进的分析技术，如果将其应用于科学同行评审，可能会彻底改变该领域。长短期记忆（LSTM）神经网络--像人脑一样识别数据中复杂模式的算法--特别是将计算机科学应用于社会和心理现象的研究。使用大量的，人口统计学上多样化的NIH R 01应用评论，分数和构建的研究部分讨论的视频，该项目正在制作分析工具，使用LSTM来捕获基于刻板印象的偏见的证据，在书面和口头讨论的拨款申请。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。