Assessing Bias and Idiosyncrasies in Elite Scientific Peer Review

同行评议是科学界正式评估新研究贡献的核心过程。科学文章的同行评议结果，特别是精英科学期刊的同行评议结果，对公众对科学知识的理解以及科学事业和科学发现的方向具有广泛的影响。与此同时，同行评议可能会表现出偏见和特质，从而产生非精英化的结果，进而限制科学进步和广泛参与，甚至可能破坏或阻碍循证政策。该项目将(i)量化、建模并理解跨多个科学领域的精英科学同行评议中偏见和特质的规模、来源和影响；（ii）促进更广泛的社区努力，使用科学方法研究同行评议；（iii）为旨在确保同行评议及其结果可靠性的新政策提供信息。该项目将(i)在两家精英普通科学期刊上生成多个综合的、匿名的同行评议数据集，涵盖多个年份和多个领域，并将其公开供研究界重用；（ii）使用这些匿名数据集定量评估精英同行评议中社会偏见和编辑特质的程度、来源和影响。这些分析将使用因果推理、统计建模、机器学习和自然语言处理的统计方法来产生最先进的效应大小和来源估计。匿名数据集将遵循NIST IR 8053去识别标准，并且匿名方案将在任何数据发布之前进行稳健评估。这些数据集将能够对提交的手稿进行定量、纵向分析，以及不同编辑和审稿人对社会人口统计学和专业属性变量的依赖，以统计评估、量化和比较手稿、编辑和领域的社会偏见和编辑特质。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Peer review is a core process by which the scientific community formally evaluates new research contributions. The outcomes of peer review for scientific articles, particularly at elite scientific journals, have broad influence on the public’s understanding of scientific knowledge, as well as on scientific careers and the directions of scientific discovery. At the same time, peer review can exhibit biases and idiosyncrasies that can produce non-meritocratic outcomes, which can, in turn, limit scientific advancement and broad participation, and can even undermine or impede evidence-based policy. This project will (i) quantify, model, and understand the magnitude, sources, and effects of biases and idiosyncrasies within elite scientific peer review across multiple scientific fields, (ii) facilitate broader community efforts in studying peer review using scientific methods, and (iii) inform new policies intended to ensure the reliability of peer review and its outcomes.This project will (i) produce multiple comprehensive, anonymized datasets of peer review at two elite general science journals, spanning multiple years and fields, and make them publicly available for reuse by the research community; and (ii) use these anonymized datasets to quantitatively assess the magnitude, source, and effects of both social biases and editorial idiosyncrasies within elite peer review. These analyses will use statistical methods from causal inference, statistical modeling, machine learning, and natural language processing to produce state-of-the-art estimates of effect sizes and sources. The anonymized datasets will adhere to the NIST IR 8053 standard for de-identification, and the anonymization scheme will be robustly evaluated prior to any data release. These data sets will enable quantitative, longitudinal analyses of submitted manuscripts, and the dependence of different editorial and reviewer recommendations on social demographic and professional attribute variables, to statistically assess, quantify, and compare social biases and editorial idiosyncrasies across manuscripts, editors, and fields.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.