Perceptions of Efficiency and Bias in Peer Review: Algorithmic versus Human Decision Making

对同行评审中的效率和偏见的看法：算法决策与人类决策

• 批准号: 2316034
• 负责人: Laurie Schintler
• 金额: $ 39.99万
• 依托单位: George Mason University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2316034&HistoricalAwards=false
• 关键词: Perceptions; Efficiency; Bias; Peer; Review

This project will develop improved methods and concepts to guide the development and application of new digital technologies that could be used in peer review processes for evaluating scientific publishing and funding outcomes. The research team will seek to compare perceptions of peer review decisions assisted by algorithms to those made by humans. The focus of the research is on the ethics and value of using algorithms in peer review. On the one hand, algorithmic peer review serves an instrumental purpose, purportedly offering the ability to make more efficient decisions. On the other hand, algorithms can produce biased and discriminatory decisions, which can raise ethical concerns about their use. This study will expand knowledge of how algorithms relate to the norms, values, and institutional imperatives that dictate how science as a human and machine endeavor should be conducted.The research team will carry out a factorial survey based on experiments in which research participants are presented with vignettes regarding human and algorithmic peer review decision making. The participants will be asked to assess the legitimacy of each scenario in light of bias and efficiency. The team will employ various techniques, principally multi-level econometric methods, to analyze data drawn from the survey and other sources, including sociology publications. The team will use an institutionalist approach to frame and delineate key concepts and relationships and to formulate research hypotheses that are empirically meaningful and theoretically appealing. The project will offer insights on measuring both pragmatic and moral legitimacy as they pertain to peer review. The project will also provide an adaptable survey tool and approach for gauging perceptions about algorithmic versus human peer review decisions and other scholarly communication activities. Accordingly, it addresses foundational issues in the philosophy of science, sociology of science and technology, and science communication.This project is jointly funded through the ER2 program by the Directorate for Social, Behavioral and Economic Sciences and the Directorate for Computer and Information Science and Engineering.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将开发改进的方法和概念，以指导新数字技术的开发和应用，这些技术可用于同行评审过程，以评估科学出版和资助成果。 研究团队将试图比较算法辅助的同行评审决策与人类决策的感知。研究的重点是在同行评议中使用算法的道德和价值。一方面，算法同行评审服务于工具性目的，据称提供了做出更有效决策的能力。另一方面，算法可能会产生有偏见和歧视性的决定，这可能会引起对其使用的道德担忧。本研究将扩展算法如何与规范、价值观和制度要求相关的知识，这些规范、价值观和制度要求决定了科学作为人类和机器的奋进应该如何进行。研究团队将基于实验进行因子调查，在实验中，研究参与者将被呈现关于人类和算法同行评审决策的小插曲。 将要求参与者根据偏见和效率评估每种设想方案的合理性。该小组将采用各种技术，主要是多层次的计量经济学方法，分析从调查和其他来源，包括社会学出版物中获得的数据。该小组将使用制度主义的方法来框架和界定关键的概念和关系，并制定研究假设，是经验上有意义的和理论上的吸引力。该项目将提供有关衡量实用和道德合法性的见解，因为它们涉及同行审查。 该项目还将提供一个适应性强的调查工具和方法，以衡量人们对算法与人类同行评审决定和其他学术交流活动的看法。因此，它解决了科学哲学，科学技术社会学和科学传播中的基础问题。该项目由社会，行为和经济科学以及计算机和信息科学与工程理事会。该奖项反映了NSF的法定使命，并通过使用基金会的知识产权进行评估，被认为值得支持。优点和更广泛的影响审查标准。