Optimizing Scientific Peer Review

科学同行评议是决定谁被发表、晋升或获奖或授予的核心过程。因此，它可能会对科学家的职业生涯和科学的方向产生巨大的影响。然而，一些研究人员已经证明，科学的同行评议可能是缓慢和低质量的。此外，一些研究量化了同行评审的偏见--例如，对某些想法的偏见--以及不一致--例如，同一项工作从不同的同行群体收到了截然不同的意见。这些问题延缓或有时截断了重要研究的传播，影响了技术发展，并最终影响了经济。该项目分析影响同行评议结果的因素，利用这些因素改进评审员的选择，开发优化评审员分配的软件，并在科学期刊、大型科学会议和大规模公开在线课程(MOOC)的现实世界背景下对结果模型进行评估。在这个项目结束时，科学界将更好地了解影响同行审查的因素和可操作的见解，以使同行审查更好。该项目的第一个组成部分量化了审查的偏见、差异、时机和质量方面的问题。这包括直接影响(例如，它们是否相互协作或引用彼此)和间接影响(例如，它们是否对同一社区做出贡献，并有望自我认同)。该项目还将偏见确定为作者和审稿人的个人特征的函数。这些方面包括年龄、性别和少数群体地位，以及它们在该领域的可见度和中心地位。同样的一般方法被用来预测审查的时间，包括选择接受审查任务。最后，本研究使用该特征集来预测评论的质量。对于给定的手稿，结果包括对每个可能的评审者的偏见和决策差异的预测，参与评审过程的可能性和时机，以及最终的评审质量。该项目的第二部分研究和开发技术，以评估潜在审稿人的特点，并利用这些推断的特点，为任何给定的手稿提出一个审查小组。这些技术优化了成本函数的期望值，该成本函数平衡了审查者选择差异(偏差和协方差)、审查时机和审查质量这三个目标。据推测，这涉及到建议由具有互补专业知识和潜在职业阶段的评审员组成的评审团，他们理解这个主题并对手稿内容感兴趣。该项目允许根据正在审议的编辑的背景、特点和职位来选择提出这些建议。最后，该项目测试了自动分配审阅者的技术，并在现实世界的应用程序中分析了过程的输出。特别是，该项目与大型期刊、科学会议和大规模开放在线课程(MOOC)组织合作。通过随机分配(当前方法与项目的算法)，该项目评估分配方法在多大程度上产生较小的评审者选择差异、更快的评审和更高质量的评审。该项目创建的软件和结果可供其他场所使用。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Scientific peer review is a central process when deciding who gets published, promoted, or awarded a prize or grant. Consequently, it may have tremendous impact on the career of scientists and the direction of science. Several researchers, however, have shown that scientific peer review can be slow and low-quality. Moreover, some studies have quantified peer review biases - e.g., prejudices against certain ideas - and inconsistencies - e.g., the same work receiving widely different opinions from different groups of peers. These problems delay or sometimes truncate the dissemination of important research, affecting technological development and ultimately the economy. This project analyzes factors that affect the outcomes of peer review, uses these to improve reviewer selection, develops software that optimizes reviewer assignments, and evaluates the resulting models in the real-world context of a scientific journal, major scientific conferences, and massive open, online courses (MOOCs). By the end of this project, the scientific community will have a better understanding of the factors that affect peer review and actionable insights to make peer review better.The first component of this project quantifies problems in bias, variance, timing, and quality of reviews. This includes direct effects (e.g., do they collaborate or cite one another) and indirect effects (e.g., do they contribute to and hopefully self-identify with the same community). The project also identifies bias as a function of personal characteristics of author and reviewer. These aspects include age, gender, and minority status, and their visibility and centrality within the field. The same general approach is used to predict the timing of reviews, including the choice to accept the review task. Lastly, the research uses this feature set to predict the quality of reviews. The result, for a given manuscript, includes prediction for each possible reviewer's biases and decision variance, likelihood and timing to participate in the review process, and ultimate review quality. The second component of this project researches and develops techniques to estimate the characteristics of potential reviewers and uses those inferred characteristics to propose, for any given manuscript, a review panel. The techniques optimize the expected value for a cost function that balances the three objectives of reviewer choice variance (bias and covariance), review timing, and review quality. Presumably, this involves suggesting panels comprised of reviewers with complementary expertise and potentially career stage, who understand the topic and are interested in the manuscripts contents. The project allows the option of making these recommendations conditional on the background, characteristics and position of the editor under consideration. Lastly, the project tests the techniques that automatically assign reviewers and analyzes the output of the process in real world applications. In particular, the project collaborates with a large journal, scientific conferences, and massive open, online course (MOOC) organizations. Through random assignments (current methods versus the project's algorithm), the project evaluates the degree to which the assignment approach produces less reviewer choice variance, faster reviews, and reviews of higher quality. The project creates software and results that can be used by other venues.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.