Collaborative Research: Comparing Single- vs. Double-Blind Review of Scientific Abstracts for Accuracy and Bias

合作研究：比较科学摘要的单盲与双盲评审的准确性和偏差

• 批准号: 1824259
• 负责人: Timothy Pleskac
• 金额: $ 10.98万
• 依托单位: University of Kansas Center for Research Inc
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-01 至 2020-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1824259&HistoricalAwards=false
• 关键词: Collaborative; Research; Comparing; Single; vs.

This research project will conduct a field experiment to compare double-blind and single-blind review processes in a high-stakes selection process. Peer-review is used to identify quality scientific work and as such is the mechanism by which the scientific community regulates itself. Science, however, does not take place in a vacuum. The possibility of bias from known characteristics such as gender, name, title, or nationality has led to frequent and recurrent calls that the evaluation of scientific work should follow a double-blind review process (where the identities of both the reviewers and the authors are withheld) rather than a single-blind review process (where the identities of the reviewer are withheld but not the authors). Yet, there are strong arguments for the use of single-blind review, including that it is difficult to truly conceal the identity of the authors and that their identities may serve as a valid cue for the quality of the work. The project will compare the outcomes from single- and double-blind reviews of talks included on the program of a national conference. The experiment will examine how well each review process predicts highly rated talks (accuracy) and whether the reviewers show preferences for author characteristics unrelated to talk quality (bias). The project will provide evidence-based recommendations for review procedures that facilitate an efficient, fair, and impartial review process. The project also will help develop methods to insure underrepresented groups have an equal and fair chance in other fields where peer review plays an important role in the research selection process, such as science, technology, engineering, and math. Graduate students will play an integral role in the conduct of this research.This research project will compare the effects of single- and double-blind review processes on conference submissions to the upcoming 2018 Annual Meeting of the Society of Judgment and Decision Making in a large-scale field study. The project will examine the degree to which characteristics such as gender, name, title, country of origin, and institution influence evaluation of conference abstracts in a high-stakes selection process. The project also will examine the impact of the use of these cues on evaluation accuracy. All abstracts submitted for a paper presentation will be subject to both single- and double-blind review. This will allow a direct comparison of the two review processes controlling for heterogeneity between papers. The accepted talks at the annual conference also will be evaluated for their quality and attendance to determine which of the two review methods better predicts these outcomes. No published study has compared actual outcomes of different review processes. Reviewer ratings will be modeled not only using the characteristics of the submitting author, but also the features of the topics in the submitted abstract, thus allowing for comparison of how different review processes impact the judgment processes of reviewers. The investigators will use the results from this large-scale field experiment to develop evidence-based recommendations for review procedures that facilitate an efficient, fair, and impartial review process.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

本研究将进行一项实地实验，比较高风险选择过程中的双盲和单盲审查过程。同行审查用于确定高质量的科学工作，因此是科学界自我管理的机制。 然而，科学并不是在真空中发生的。由于性别、姓名、职称或国籍等已知特征可能存在偏见，因此经常有人呼吁对科学工作的评价应遵循双盲审查过程（审查者和作者的身份均被保留），而不是单盲审查过程（审查者的身份被保留，但作者的身份不被保留）。然而，有强有力的论据支持使用单盲审查，包括很难真正隐藏作者的身份，他们的身份可以作为工作质量的有效线索。该项目将比较全国会议计划中包括的单盲和双盲审查的结果。实验将检验每个评论过程预测高评分谈话的准确性，以及评论者是否对与谈话质量无关的作者特征表现出偏好（偏见）。该项目将为审查程序提供基于证据的建议，以促进高效、公平和公正的审查过程。该项目还将帮助制定方法，以确保代表性不足的群体在同行评议在研究选择过程中发挥重要作用的其他领域，如科学，技术，工程，和数学。研究生将在这项研究的进行中发挥不可或缺的作用。这项研究项目将比较单一和双重的影响，在一项大规模的实地研究中，对即将举行的2018年判断与决策学会年会的会议提交材料进行盲审。该项目将研究性别、姓名、职称、原籍国和机构等特征在高风险选择过程中对会议摘要评价的影响程度。该项目还将研究使用这些线索对评估准确性的影响。所有提交的论文摘要将接受单盲和双盲审查。这将允许直接比较两个评审过程，控制论文之间的异质性。年会上接受的演讲也将被评估其质量和出席率，以确定两种审查方法中哪一种更好地预测这些结果。没有发表的研究比较了不同审查过程的实际结果。审稿人评分不仅将使用提交作者的特征，而且还将使用提交摘要中主题的特征进行建模，从而可以比较不同的评审过程如何影响审稿人的判断过程。调查人员将利用这一大规模实地实验的结果，为审查程序提出基于证据的建议，以促进有效、公平和公正的审查过程。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。