SCISIPBIO: SBP: Collaborative Research: Reliability, Risk Aversion, and Bias in Grant Peer Review

SCISIPBIO：SBP：协作研究：资助同行评审中的可靠性、风险规避和偏见

• 批准号: 1951132
• 负责人: Stephen Gallo
• 金额: $ 10.96万
• 依托单位: American Institute of Biological Sciences
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-06-01 至 2022-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1951132&HistoricalAwards=false
• 关键词: SCISIPBIO; SBP; Collaborative; Research; Reliability

Agencies that fund scientific research use peer review to identify the most meritorious research projects for support. Peer review involves multiple expert scientists evaluating applications based on pre-determined criteria (e.g., the quality of the methodology, the qualifications of the scientists, etc.) and assigning it a score. As research funding is highly competitive, peer reviewers must identify the most outstanding projects among many of those deserving funding. Peer reviewers’ judgments, however, are subject to individual biases and preferences. Currently, little research exists on peer reviewer preferences and how these may influence their judgments. In this project, peer review is considered as a process of evaluating and weighing project risks and benefits. The hypothesis is that reviewers’ attitude towards risk, i.e., risk preference, will influence their judgments and scores. Science can proceed in smaller steps or in larger leaps. Risk is important to progress in science: innovative and novel projects may be associated with higher risk, and result in great achievement or failure. Therefore, peer reviewer risk preference may influence the rate of scientific progress. This project involves an experiment in which peer reviewers judge the merits of projects that vary in terms of risk. The information from this project will inform peer reviewer training and evaluation criteria. In addition, a module for undergraduate science classes will educate students on peer review and potential biases.The hypothesis is tested in this project by having peer reviewers evaluate and score a set of fictitious NIH-style overall impact statements (i.e., summary critiques) for an R01-type research grant application. Different versions of the impact statement are created to systematically reflect different sources of risk (methodology versus investigator) and levels of risk (high/low) associated with the application. The text is varied to reflect either female or male applicants. Peer reviewer participants also complete questions about demographic information, expertise, self-assessed level of bias, and a measure of risk preference. The factors associated with reviewer demographics and risk preference are then tested as potential predictors of the reviewers’ scores; furthermore, the interaction of these predictors with different sources and levels of risk is examined. Finally, the effect of gender on reviewers’ scores is examined, and how gender interacts with reviewer preference and type and level of risk. The results will account for more of the variability in reviewers’ scores than is evident in the extant research and potentially inform efforts to orient peer reviewers regarding risk preference and biases.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

资助科学研究的机构使用同行评议来确定最值得支持的研究项目。同行评议涉及多位专家科学家根据预先确定的标准（例如，方法的质量，科学家的资格等）评估应用程序并给它打分。由于研究经费的竞争非常激烈，同行审稿人必须在众多值得资助的项目中找出最杰出的项目。然而，同行审稿人的判断受制于个人偏见和偏好。目前，关于同行评议人的偏好以及这些偏好如何影响他们的判断的研究很少。在这个项目中，同行评审被认为是一个评估和权衡项目风险和收益的过程。假设审稿人对风险的态度，即风险偏好，会影响他们的判断和评分。科学可以小步前进，也可以大跃进。风险对科学进步很重要：创新和新颖的项目可能与更高的风险相关，并导致巨大的成就或失败。因此，同行审稿人的风险偏好可能会影响科学进步的速度。这个项目涉及到一个实验，在这个实验中，同行评审人员根据风险的不同来判断项目的优点。来自这个项目的信息将为同行审稿人的培训和评估标准提供信息。此外，本科理科课程的一个模块将教育学生同行评议和潜在偏见。在这个项目中，通过让同行评审员对一组虚构的nih风格的总体影响陈述（即总结评论）进行评估和评分，对r01类型的研究资助申请进行假设测试。创建不同版本的影响陈述是为了系统地反映与应用程序相关的不同风险来源（方法与研究者）和风险水平（高/低）。为了反映女性或男性申请者的情况，简历内容各不相同。同行评议参与者还完成关于人口统计信息、专业知识、自我评估的偏见水平和风险偏好的测量问题。然后测试与审稿人人口统计和风险偏好相关的因素作为审稿人得分的潜在预测因素；此外，这些预测因素与不同的来源和风险水平的相互作用进行了检查。最后，研究了性别对审稿人得分的影响，以及性别如何与审稿人偏好、风险类型和水平相互作用。与现有研究相比，这些结果将更多地解释审稿人分数的可变性，并可能为同行审稿人在风险偏好和偏见方面的定位提供信息。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。