SCISIPBIO: SBP: Collaborative Research: Reliability, Risk Aversion, and Bias in Grant Peer Review

资助科学研究的机构利用同行评审来确定最值得支持的研究项目。同行评审涉及多位专家科学家根据预先确定的标准（例如，方法的质量、科学家的资格等）并给它打分由于研究经费的竞争非常激烈，同行评审员必须在许多值得资助的项目中找出最优秀的项目。然而，同行评审员的判断会受到个人偏见和偏好的影响。目前，很少有研究存在同行评议员的偏好，以及如何这些可能会影响他们的判断。在本项目中，同行评审被视为评估和权衡项目风险和收益的过程。假设是，审查员对风险的态度，即，风险偏好会影响他们的判断和得分。科学可以小步前进，也可以大步前进。风险对科学进步很重要：创新和新颖的项目可能与更高的风险相关，并导致巨大的成就或失败。因此，同行评审者的风险偏好可能会影响科学进步的速度。这个项目涉及一个实验，在这个实验中，同行评审员根据风险来判断项目的优点。该项目提供的信息将为同行审评员培训和评价标准提供信息。此外，本科科学课程的一个模块将教育学生同行评审和潜在的偏见。在这个项目中，通过让同行评审者评估和评分一组虚构的NIH风格的整体影响陈述（即，摘要评论）的R 01型研究资助申请。创建不同版本的影响说明，以系统地反映与应用程序相关的不同风险来源（方法与调查人员）和风险水平（高/低）。案文有所不同，以反映女性或男性申请人。同行评审参与者还完成了关于人口统计信息，专业知识，自我评估的偏见水平和风险偏好的衡量标准的问题。与审查人口统计学和风险偏好相关的因素，然后测试作为潜在的预测的审查员的分数，此外，这些预测与不同的来源和风险水平的相互作用进行了检查。最后，性别对审稿人评分的影响进行了研究，以及性别如何与审稿人偏好和风险类型和水平相互作用。结果将占更多的变化，在审稿人的分数比是明显的，在现存的研究和潜在的通知努力，以指导同行评审员对风险偏好和biases.This奖项反映了NSF的法定使命，并已被认为是值得的支持，通过评估使用基金会的智力价值和更广泛的影响审查标准。

Agencies that fund scientific research use peer review to identify the most meritorious research projects for support. Peer review involves multiple expert scientists evaluating applications based on pre-determined criteria (e.g., the quality of the methodology, the qualifications of the scientists, etc.) and assigning it a score. As research funding is highly competitive, peer reviewers must identify the most outstanding projects among many of those deserving funding. Peer reviewers’ judgments, however, are subject to individual biases and preferences. Currently, little research exists on peer reviewer preferences and how these may influence their judgments. In this project, peer review is considered as a process of evaluating and weighing project risks and benefits. The hypothesis is that reviewers’ attitude towards risk, i.e., risk preference, will influence their judgments and scores. Science can proceed in smaller steps or in larger leaps. Risk is important to progress in science: innovative and novel projects may be associated with higher risk, and result in great achievement or failure. Therefore, peer reviewer risk preference may influence the rate of scientific progress. This project involves an experiment in which peer reviewers judge the merits of projects that vary in terms of risk. The information from this project will inform peer reviewer training and evaluation criteria. In addition, a module for undergraduate science classes will educate students on peer review and potential biases.The hypothesis is tested in this project by having peer reviewers evaluate and score a set of fictitious NIH-style overall impact statements (i.e., summary critiques) for an R01-type research grant application. Different versions of the impact statement are created to systematically reflect different sources of risk (methodology versus investigator) and levels of risk (high/low) associated with the application. The text is varied to reflect either female or male applicants. Peer reviewer participants also complete questions about demographic information, expertise, self-assessed level of bias, and a measure of risk preference. The factors associated with reviewer demographics and risk preference are then tested as potential predictors of the reviewers’ scores; furthermore, the interaction of these predictors with different sources and levels of risk is examined. Finally, the effect of gender on reviewers’ scores is examined, and how gender interacts with reviewer preference and type and level of risk. The results will account for more of the variability in reviewers’ scores than is evident in the extant research and potentially inform efforts to orient peer reviewers regarding risk preference and biases.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.