NSF-BSF: RI: Small: Mechanisms and Algorithms for Improving Peer Selection

同行评议、评估和选择的过程是现代科学的一个基本方面。世界各地的资助机构和学术出版物聘请专家审查和选择最好的科学资助和出版。从一组对等体中评估和选择最佳的过程是更普遍的问题。例如，一个专业协会可能希望根据所有成员的意见给其成员的一个子集颁奖，或者一个大规模开放式在线课程（MOOC）的讲师可能希望众包分级，或者一个营销公司可能根据同行评估从小组头脑风暴会议中选择想法。在所有这些情况下，我们希望选出一小部分被社区本身认为是最好的获奖者-包括那些正在竞争和可能有利益冲突的人。这个问题，被称为同行选择问题，是本研究的重点。在同行选择设置之间可能有竞争的优先级和固有的偏见的一套审查，它是必要的开发方法和算法，调整个人的奖励审查的总体目标选择最好的一套。这个项目的智力价值在于扩大我们的理解和开发新的算法的过程中的同行评价和同行选择。在使用同行评议的领域，利益冲突和同行选择偏见被认为是更广泛参与科学的障碍。该项目将产生广泛的影响，通过过滤一些审稿人无意识的偏见和利益冲突，使同行评审过程更加健全，以公平选择，从而为研究和教育提供更好的基础设施。该项目将通过四个具体目标实现扩大我们的知识和建立同行评估和选择机制的目标。第一个目的是开发新的指标，通过定义规范和定量的属性，允许精确地描述功能的同行评价和选择过程中的同行选择机制的评价。第二个目标是开发分布式对等选择机制，能够使用，而不需要一个集中式控制器。该项目将开发工具，以了解这些机制在这种分布式环境中的行为，以及为这种环境带来的独特挑战创造新机制的机会。第三个目的是发展我们对同行选择的多阶段同行评价的理解。许多学术会议，期刊，甚至一些NSF计划的滚动审查周期的动机，有必要调查同行评价和选择机制的属性时，审查（评价）可能会传播特定的选择设置。最后的目的是激励同行选择的努力：在公正的经典社会选择属性之间存在着根本的紧张关系，即，代理人可能不会影响他们自己被接受的概率，并为评审者提供激励，使其在同行评价过程中投入精力。该项目将开发一个工具包的机制，使系统设计者能够合理地选择之间的信息量的代理知道，努力的激励措施和潜在的恶意行为的权衡。这个奖项反映了NSF的法定使命，并已被认为是值得的支持，通过评估使用基金会的智力价值和更广泛的影响审查标准。

The process of peer review, evaluation, and selection is a fundamental aspect of modern science. Funding bodies and academic publications around the world employ experts to review and select the best science for funding and publication. The process of evaluating and selecting the best from among a group of peers is much more general problem. For example, a professional society may want to give a subset of its members awards based on the opinions of all members or an instructor for a Massive Open Online Course (MOOC) may want to crowdsource grading or a marketing company may select ideas from group brainstorming sessions based on peer evaluation. In all of these settings, we wish to select a small set of winners that are judged to be the best by the community itself -- which includes those who are competing and who may have conflict of interests. This problem, known as the peer selection problem, is the focus of this research. Within a peer selection setting there may be competing priorities and inherent biases amongst the set of reviewers, and it is necessary to develop methods and algorithms that align the individual incentives of reviewers with the overall goal of selecting the best set. The intellectual merit of this project lies in expanding our understanding and developing novel algorithms for the process of peer evaluation and peer selection. Within the fields that use peer review, conflict of interest and peer selection bias have been cited as an impediment for broader participation in the science. This project will have broad impact through making the peer review process more robust to equitable selection by filtering some reviewers’ unconscious biases and conflict of interest thus resulting in a better infrastructure for research and education.The project will achieve its goal of expanding our knowledge and building mechanisms for peer evaluation and selection through four specific aims. The first aim is to develop novel metrics for the evaluation of peer selection mechanisms by defining both normative and quantitative properties that allow to precisely describe features of the peer evaluation and selection process. The second aim is to develop distributed peer selection mechanisms that are able to be used without requiring a centralized controller. This project will develop tools to understand how these mechanisms behave in this distributed setting as well as opportunities to create novel mechanisms for the unique challenges this setting poses. The third aim is to develop our understanding of multi-stage peer evaluation for peer selection. Motivated by the rolling review cycle of many academic conferences, journals, and even some NSF programs, there is a need to investigate the properties of peer evaluation and selection mechanisms when reviews (evaluations) may propagate between specific selection settings. The final aim is to incentivize effort in peer selection: There is a fundamental tension between the classic social choice properties of impartiality, i.e., an agent may not affect their own probability of getting accepted, and provide incentives for reviewers to invest effort in the peer evaluation process. This project will develop a tool kit of mechanisms that allow system designers to rationally choose tradeoffs between the amount of information an agent knows, incentives for effort, and potential for malicious behavior.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.