Assessing Bias and Idiosyncrasies in Elite Scientific Peer Review

同行评议是科学界正式评估新的研究贡献的核心过程。同行评议的结果对公众对科学知识的理解，以及对科学事业和科学发现的方向有着广泛的影响，特别是在精英科学期刊上。与此同时，同行审查可能会表现出偏见和特质，可能会产生非任人唯贤的结果，进而可能限制科学进步和广泛参与，甚至可能破坏或阻碍以证据为基础的政策。这个项目将(I)对跨多个科学领域的精英科学同行评审中的偏见和特质的规模、来源和影响进行量化、建模和理解，(Ii)促进更广泛的社区在使用科学方法研究同行评审方面的努力，以及(Iii)提供旨在确保同行评审及其结果的可靠性的新政策。该项目将(I)在两个跨越多年和多个领域的精英普通科学期刊上生成多个全面的匿名同行评审数据集，并将其公开供研究界重复使用；和(Ii)使用这些匿名的数据集来定量评估精英同行评议中的社会偏见和编辑特质的规模、来源和影响。这些分析将使用来自因果推理、统计建模、机器学习和自然语言处理的统计方法来产生对影响大小和来源的最先进的估计。匿名化的数据集将遵循NIST IR 8053标准进行去身份识别，并且匿名化方案将在任何数据发布之前进行强有力的评估。这些数据集将能够对提交的手稿进行定量的纵向分析，以及不同的编辑和审稿人建议对社会人口和专业属性变量的依赖，以统计评估、量化和比较跨手稿、编辑和领域的社会偏见和编辑特质。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Peer review is a core process by which the scientific community formally evaluates new research contributions. The outcomes of peer review for scientific articles, particularly at elite scientific journals, have broad influence on the public’s understanding of scientific knowledge, as well as on scientific careers and the directions of scientific discovery. At the same time, peer review can exhibit biases and idiosyncrasies that can produce non-meritocratic outcomes, which can, in turn, limit scientific advancement and broad participation, and can even undermine or impede evidence-based policy. This project will (i) quantify, model, and understand the magnitude, sources, and effects of biases and idiosyncrasies within elite scientific peer review across multiple scientific fields, (ii) facilitate broader community efforts in studying peer review using scientific methods, and (iii) inform new policies intended to ensure the reliability of peer review and its outcomes.This project will (i) produce multiple comprehensive, anonymized datasets of peer review at two elite general science journals, spanning multiple years and fields, and make them publicly available for reuse by the research community; and (ii) use these anonymized datasets to quantitatively assess the magnitude, source, and effects of both social biases and editorial idiosyncrasies within elite peer review. These analyses will use statistical methods from causal inference, statistical modeling, machine learning, and natural language processing to produce state-of-the-art estimates of effect sizes and sources. The anonymized datasets will adhere to the NIST IR 8053 standard for de-identification, and the anonymization scheme will be robustly evaluated prior to any data release. These data sets will enable quantitative, longitudinal analyses of submitted manuscripts, and the dependence of different editorial and reviewer recommendations on social demographic and professional attribute variables, to statistically assess, quantify, and compare social biases and editorial idiosyncrasies across manuscripts, editors, and fields.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.