NSF-BSF: RI: Small: Mechanisms and Algorithms for Improving Peer Selection

NSF-BSF：RI：小型：改进同行选择的机制和算法

• 批准号: 2134857
• 负责人: Nicholas Mattei
• 金额: $ 30.89万
• 依托单位: Tulane University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2024-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2134857&HistoricalAwards=false
• 关键词: NSF; BSF; RI; Small; Mechanisms

The process of peer review, evaluation, and selection is a fundamental aspect of modern science. Funding bodies and academic publications around the world employ experts to review and select the best science for funding and publication. The process of evaluating and selecting the best from among a group of peers is much more general problem. For example, a professional society may want to give a subset of its members awards based on the opinions of all members or an instructor for a Massive Open Online Course (MOOC) may want to crowdsource grading or a marketing company may select ideas from group brainstorming sessions based on peer evaluation. In all of these settings, we wish to select a small set of winners that are judged to be the best by the community itself -- which includes those who are competing and who may have conflict of interests. This problem, known as the peer selection problem, is the focus of this research. Within a peer selection setting there may be competing priorities and inherent biases amongst the set of reviewers, and it is necessary to develop methods and algorithms that align the individual incentives of reviewers with the overall goal of selecting the best set. The intellectual merit of this project lies in expanding our understanding and developing novel algorithms for the process of peer evaluation and peer selection. Within the fields that use peer review, conflict of interest and peer selection bias have been cited as an impediment for broader participation in the science. This project will have broad impact through making the peer review process more robust to equitable selection by filtering some reviewers’ unconscious biases and conflict of interest thus resulting in a better infrastructure for research and education.The project will achieve its goal of expanding our knowledge and building mechanisms for peer evaluation and selection through four specific aims. The first aim is to develop novel metrics for the evaluation of peer selection mechanisms by defining both normative and quantitative properties that allow to precisely describe features of the peer evaluation and selection process. The second aim is to develop distributed peer selection mechanisms that are able to be used without requiring a centralized controller. This project will develop tools to understand how these mechanisms behave in this distributed setting as well as opportunities to create novel mechanisms for the unique challenges this setting poses. The third aim is to develop our understanding of multi-stage peer evaluation for peer selection. Motivated by the rolling review cycle of many academic conferences, journals, and even some NSF programs, there is a need to investigate the properties of peer evaluation and selection mechanisms when reviews (evaluations) may propagate between specific selection settings. The final aim is to incentivize effort in peer selection: There is a fundamental tension between the classic social choice properties of impartiality, i.e., an agent may not affect their own probability of getting accepted, and provide incentives for reviewers to invest effort in the peer evaluation process. This project will develop a tool kit of mechanisms that allow system designers to rationally choose tradeoffs between the amount of information an agent knows, incentives for effort, and potential for malicious behavior.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

同行评审、评价和选择的过程是现代科学的一个基本方面。世界各地的资助机构和学术出版物聘请专家审查和选择最好的科学资助和出版。从一群同伴中评估和选择最佳的过程是一个更普遍的问题。例如，一个专业协会可能希望根据所有成员的意见给其成员的一部分奖励，或者一个大规模开放在线课程（MOOC）的讲师可能想要众包评分，或者一个营销公司可能会根据同行评估从小组头脑风暴会议中选择想法。在所有这些设置中，我们希望选择一小部分被社区本身认为是最好的赢家——包括那些竞争和可能有利益冲突的人。这个问题被称为同伴选择问题，是本研究的重点。在同行选择设置中，审稿人之间可能存在竞争优先级和固有偏见，有必要开发方法和算法，使审稿人的个人动机与选择最佳集合的总体目标保持一致。这个项目的智力价值在于扩展了我们对同行评估和同行选择过程的理解并开发了新的算法。在使用同行评议的领域，利益冲突和同行选择偏见被认为是更广泛参与科学的障碍。该项目将产生广泛的影响，通过过滤一些审稿人无意识的偏见和利益冲突，使同行评审过程更加稳健，从而实现公平的选择，从而为研究和教育提供更好的基础设施。该项目将通过四个具体目标来实现扩大我们的知识和建立同行评估和选择机制的目标。第一个目标是通过定义规范和定量的属性，从而精确描述同行评估和选择过程的特征，为同行选择机制的评估制定新的度量标准。第二个目标是开发不需要集中式控制器就可以使用的分布式对等选择机制。该项目将开发工具来了解这些机制在这种分布式环境中的行为，并为这种环境带来的独特挑战创造新的机制。第三个目标是发展我们对同伴选择的多阶段同伴评估的理解。在许多学术会议、期刊甚至一些NSF项目的滚动评审周期的推动下，当评审（评价）可能在特定的选择设置之间传播时，有必要研究同行评审和选择机制的特性。最终目标是激励同伴选择的努力：在公正的经典社会选择属性之间存在着根本的紧张关系，即代理可能不会影响自己被接受的概率，并为审稿人在同伴评估过程中投入努力提供激励。该项目将开发一个机制工具包，允许系统设计者在代理知道的信息量、努力的激励和恶意行为的可能性之间合理地选择权衡。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments

• DOI: 10.1016/j.artint.2022.103843
• 发表时间: 2022-12
• 期刊: Artif. Intell.
• 作者: Omer Lev;Nicholas Mattei;P. Turrini;Stanislav Zhydkov

Who Reviews The Reviewers? A Multi-Level Jury Problem

• 发表时间: 2022-11
• 作者: Ben Abramowitz;Nicholas Mattei

Mitigating Skewed Bidding for Conference Paper Assignment

• DOI: 10.48550/arxiv.2303.00435
• 发表时间: 2023-03
• 作者: Inbal Rozencweig;R. Meir;Nick Mattei;Ofra Amir