Optimizing Scientific Peer Review

科学同行评议是决定谁的论文发表、晋升、获奖或获得资助的核心过程。因此，它可能会对科学家的事业和科学的方向产生巨大的影响。然而，一些研究人员已经表明，科学同行评议可能是缓慢和低质量的。此外，一些研究还量化了同行评议的偏见（例如，对某些观点的偏见）和不一致性（例如，同一项工作从不同的同行群体中得到截然不同的意见）。这些问题延迟或有时中断重要研究的传播，影响技术发展并最终影响经济。本项目分析影响同行评议结果的因素，利用这些因素来改进审稿人的选择，开发优化审稿人分配的软件，并在科学期刊、重大科学会议和大规模开放在线课程（MOOCs）的现实环境中评估结果模型。在本项目结束时，科学界将对影响同行评议的因素有更好的了解，并有可操作的见解，使同行评议更好。这个项目的第一个组成部分量化了偏差、方差、时间和评审质量方面的问题。这包括直接影响（例如，他们是否合作或相互引用）和间接影响（例如，他们是否为同一个社区做出贡献并希望自我认同）。该项目还将偏见确定为作者和审稿人个人特征的函数。这些方面包括年龄、性别和少数民族地位，以及他们在该领域的知名度和中心地位。同样的一般方法用于预测评审的时间，包括选择接受评审任务。最后，该研究使用该特征集来预测评论的质量。对于给定的手稿，结果包括对每个可能的审稿人的偏见和决策差异，参与审稿过程的可能性和时间，以及最终审稿质量的预测。该项目的第二个组成部分研究和开发了评估潜在审稿人特征的技术，并使用这些推断的特征为任何给定的手稿提出一个审稿人小组。这些技术优化了成本函数的期望值，该函数平衡了审稿人选择方差（偏差和协方差）、审查时间和审查质量这三个目标。据推测，这包括建议由具有互补专业知识和潜在职业阶段的审稿人组成的小组，他们了解主题并对手稿内容感兴趣。该项目允许选择根据所考虑的编辑的背景、特点和职位提出这些建议。最后，该项目测试了在实际应用程序中自动分配审阅者和分析流程输出的技术。特别是，该项目与大型期刊、科学会议和大规模开放在线课程（MOOC）组织合作。通过随机分配（当前方法相对于项目的算法），项目评估分配方法产生更少的审阅者选择差异、更快的审阅和更高质量的审阅的程度。该项目创建的软件和结果可以被其他场所使用。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Scientific peer review is a central process when deciding who gets published, promoted, or awarded a prize or grant. Consequently, it may have tremendous impact on the career of scientists and the direction of science. Several researchers, however, have shown that scientific peer review can be slow and low-quality. Moreover, some studies have quantified peer review biases - e.g., prejudices against certain ideas - and inconsistencies - e.g., the same work receiving widely different opinions from different groups of peers. These problems delay or sometimes truncate the dissemination of important research, affecting technological development and ultimately the economy. This project analyzes factors that affect the outcomes of peer review, uses these to improve reviewer selection, develops software that optimizes reviewer assignments, and evaluates the resulting models in the real-world context of a scientific journal, major scientific conferences, and massive open, online courses (MOOCs). By the end of this project, the scientific community will have a better understanding of the factors that affect peer review and actionable insights to make peer review better.The first component of this project quantifies problems in bias, variance, timing, and quality of reviews. This includes direct effects (e.g., do they collaborate or cite one another) and indirect effects (e.g., do they contribute to and hopefully self-identify with the same community). The project also identifies bias as a function of personal characteristics of author and reviewer. These aspects include age, gender, and minority status, and their visibility and centrality within the field. The same general approach is used to predict the timing of reviews, including the choice to accept the review task. Lastly, the research uses this feature set to predict the quality of reviews. The result, for a given manuscript, includes prediction for each possible reviewer's biases and decision variance, likelihood and timing to participate in the review process, and ultimate review quality. The second component of this project researches and develops techniques to estimate the characteristics of potential reviewers and uses those inferred characteristics to propose, for any given manuscript, a review panel. The techniques optimize the expected value for a cost function that balances the three objectives of reviewer choice variance (bias and covariance), review timing, and review quality. Presumably, this involves suggesting panels comprised of reviewers with complementary expertise and potentially career stage, who understand the topic and are interested in the manuscripts contents. The project allows the option of making these recommendations conditional on the background, characteristics and position of the editor under consideration. Lastly, the project tests the techniques that automatically assign reviewers and analyzes the output of the process in real world applications. In particular, the project collaborates with a large journal, scientific conferences, and massive open, online course (MOOC) organizations. Through random assignments (current methods versus the project's algorithm), the project evaluates the degree to which the assignment approach produces less reviewer choice variance, faster reviews, and reviews of higher quality. The project creates software and results that can be used by other venues.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.