SCISIPBIO: SBP: Collaborative Research: Reliability, Risk Aversion, and Bias in Grant Peer Review

资助科学研究的机构使用同行评审来确定最有价值的研究项目以获得支持。同行评审涉及多名专家科学家根据预先确定的标准(例如，方法的质量、科学家的资质等)对申请进行评估。并给它打分。由于研究资金的竞争非常激烈，同行评审员必须在许多值得资助的项目中找出最杰出的项目。然而，同行评审员的判断受到个人偏见和偏好的影响。目前，关于同行评审者的偏好以及这些偏好如何影响他们的判断的研究很少。在本项目中，同行评审被视为评估和权衡项目风险和收益的过程。假设评价者对风险的态度，即风险偏好，会影响他们的判断和分数。科学可以更小的步骤，也可以更大的飞跃。风险对科学进步很重要：创新和新奇的项目可能伴随着更高的风险，并导致巨大的成就或失败。因此，同行评审员的风险偏好可能会影响科学进步的速度。这个项目涉及一个实验，在这个实验中，同行评审员判断风险不同的项目的优点。来自该项目的信息将为同行评审员培训和评估标准提供信息。此外，本科生科学课程的一个模块将教育学生关于同行评议和潜在的偏见。在这个项目中，通过让同行评审员评估和评分一组虚构的NIH风格的总体影响陈述(即摘要评论)来检验这个项目的假设。创建了不同版本的影响说明书，以系统地反映与应用程序相关的不同风险来源(方法和调查者)和风险级别(高/低)。文本不同，以反映女性或男性申请者。同行评审员还完成了关于人口统计信息、专业知识、自我评估的偏见水平和风险偏好衡量标准的问题。与评审者人口统计和风险偏好相关的因素随后被测试为评价者得分的潜在预测因素；此外，还考察了这些预测因素与不同风险来源和风险水平的交互作用。最后，考察了性别对评价者得分的影响，以及性别与评价者偏好、风险类型和水平之间的相互作用。这一结果将比现有研究中更明显地解释评审员分数的变异性，并可能为指导同行评审员有关风险偏好和偏见的努力提供参考。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Agencies that fund scientific research use peer review to identify the most meritorious research projects for support. Peer review involves multiple expert scientists evaluating applications based on pre-determined criteria (e.g., the quality of the methodology, the qualifications of the scientists, etc.) and assigning it a score. As research funding is highly competitive, peer reviewers must identify the most outstanding projects among many of those deserving funding. Peer reviewers’ judgments, however, are subject to individual biases and preferences. Currently, little research exists on peer reviewer preferences and how these may influence their judgments. In this project, peer review is considered as a process of evaluating and weighing project risks and benefits. The hypothesis is that reviewers’ attitude towards risk, i.e., risk preference, will influence their judgments and scores. Science can proceed in smaller steps or in larger leaps. Risk is important to progress in science: innovative and novel projects may be associated with higher risk, and result in great achievement or failure. Therefore, peer reviewer risk preference may influence the rate of scientific progress. This project involves an experiment in which peer reviewers judge the merits of projects that vary in terms of risk. The information from this project will inform peer reviewer training and evaluation criteria. In addition, a module for undergraduate science classes will educate students on peer review and potential biases.The hypothesis is tested in this project by having peer reviewers evaluate and score a set of fictitious NIH-style overall impact statements (i.e., summary critiques) for an R01-type research grant application. Different versions of the impact statement are created to systematically reflect different sources of risk (methodology versus investigator) and levels of risk (high/low) associated with the application. The text is varied to reflect either female or male applicants. Peer reviewer participants also complete questions about demographic information, expertise, self-assessed level of bias, and a measure of risk preference. The factors associated with reviewer demographics and risk preference are then tested as potential predictors of the reviewers’ scores; furthermore, the interaction of these predictors with different sources and levels of risk is examined. Finally, the effect of gender on reviewers’ scores is examined, and how gender interacts with reviewer preference and type and level of risk. The results will account for more of the variability in reviewers’ scores than is evident in the extant research and potentially inform efforts to orient peer reviewers regarding risk preference and biases.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.