Collaborative Research: Comparing Single- vs. Double-Blind Review of Scientific Abstracts for Accuracy and Bias

这项研究项目将进行现场实验，比较高风险选择过程中的双盲和单盲审查过程。同行评议是用来确定高质量的科学工作的，因此也是科学界自我管理的机制。然而，科学并不是在真空中发生的。由于可能存在性别、姓名、职称或国籍等已知特征的偏见，经常有人呼吁对科学工作的评价应遵循双盲审查程序(不公布评审者和作者的身份)，而不是单盲审查程序(不公布评审者的身份，而不是作者)。然而，使用单盲审查有很强的理由，包括很难真正掩盖作者的身份，他们的身份可能是作品质量的有效线索。该项目将比较对全国会议议程上的会谈进行单盲和双盲审查的结果。这项实验将检验每个评论过程预测高评分演讲的准确性(准确性)，以及评论者是否表现出对与演讲质量无关的作者特征的偏好(偏见)。该项目将为审查程序提供基于证据的建议，以促进高效、公平和公正的审查过程。该项目还将帮助开发方法，以确保代表不足的群体在同行审查在研究选择过程中发挥重要作用的其他领域有平等和公平的机会，如科学、技术、工程和数学。研究生将在这项研究中发挥不可或缺的作用。这项研究项目将在一项大规模的实地研究中，比较单盲和双盲审查过程对即将举行的判断与决策学会2018年年会的会议提交的影响。该项目将审查在高风险的选择过程中，性别、姓名、标题、原籍国和机构等特征对会议摘要评估的影响程度。该项目还将检查这些线索的使用对评估准确性的影响。所有提交论文报告的摘要都将接受单盲和双盲审查。这将允许对控制论文之间异质性的两个审查过程进行直接比较。还将对年度会议上接受的会谈的质量和出席情况进行评估，以确定两种审查方法中哪一种更能预测这些结果。没有发表的研究对不同审查过程的实际结果进行了比较。评审者评分将不仅使用提交作者的特征，而且还使用提交摘要中主题的特征进行建模，从而能够比较不同的审查过程如何影响审评者的判断过程。调查人员将利用这一大规模现场试验的结果，为审查程序制定基于证据的建议，以促进高效、公平和公正的审查过程。这一裁决反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This research project will conduct a field experiment to compare double-blind and single-blind review processes in a high-stakes selection process. Peer-review is used to identify quality scientific work and as such is the mechanism by which the scientific community regulates itself. Science, however, does not take place in a vacuum. The possibility of bias from known characteristics such as gender, name, title, or nationality has led to frequent and recurrent calls that the evaluation of scientific work should follow a double-blind review process (where the identities of both the reviewers and the authors are withheld) rather than a single-blind review process (where the identities of the reviewer are withheld but not the authors). Yet, there are strong arguments for the use of single-blind review, including that it is difficult to truly conceal the identity of the authors and that their identities may serve as a valid cue for the quality of the work. The project will compare the outcomes from single- and double-blind reviews of talks included on the program of a national conference. The experiment will examine how well each review process predicts highly rated talks (accuracy) and whether the reviewers show preferences for author characteristics unrelated to talk quality (bias). The project will provide evidence-based recommendations for review procedures that facilitate an efficient, fair, and impartial review process. The project also will help develop methods to insure underrepresented groups have an equal and fair chance in other fields where peer review plays an important role in the research selection process, such as science, technology, engineering, and math. Graduate students will play an integral role in the conduct of this research.This research project will compare the effects of single- and double-blind review processes on conference submissions to the upcoming 2018 Annual Meeting of the Society of Judgment and Decision Making in a large-scale field study. The project will examine the degree to which characteristics such as gender, name, title, country of origin, and institution influence evaluation of conference abstracts in a high-stakes selection process. The project also will examine the impact of the use of these cues on evaluation accuracy. All abstracts submitted for a paper presentation will be subject to both single- and double-blind review. This will allow a direct comparison of the two review processes controlling for heterogeneity between papers. The accepted talks at the annual conference also will be evaluated for their quality and attendance to determine which of the two review methods better predicts these outcomes. No published study has compared actual outcomes of different review processes. Reviewer ratings will be modeled not only using the characteristics of the submitting author, but also the features of the topics in the submitted abstract, thus allowing for comparison of how different review processes impact the judgment processes of reviewers. The investigators will use the results from this large-scale field experiment to develop evidence-based recommendations for review procedures that facilitate an efficient, fair, and impartial review process.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.