NSF-BSF: RI: Small: Mechanisms and Algorithms for Improving Peer Selection

同行评审、评估和选择的过程是现代科学的一个基本方面。世界各地的资助机构和学术出版物聘请专家审查和选择最好的科学来资助和出版。从一组同行中评估和选择最好的过程是一个更普遍的问题。例如，专业协会可能想要根据所有成员的意见给其成员的子集颁奖，或者大型在线公开课(MOOC)的讲师可能想要众包评分，或者营销公司可能会根据同行评估从小组集思广益会议中选择想法。在所有这些情况下，我们希望选出一小部分被社区本身评为最佳的获胜者--其中包括那些参与竞争的人和可能存在利益冲突的人。这一问题被称为同伴选择问题，是本研究的重点。在同行选择设置中，在评审者集合之间可能存在竞争的优先级和固有的偏见，并且有必要开发使评审者的个人激励与选择最佳集合的总体目标相一致的方法和算法。这个项目的智力价值在于扩大了我们的理解，并为同行评估和同行选择过程开发了新的算法。在使用同行评审的领域内，利益冲突和同行选择偏见被认为是更广泛地参与科学的障碍。这一项目将产生广泛的影响，通过过滤一些审评者的无意识偏见和利益冲突，使同行评议程序更有力地促进公平选择，从而为研究和教育建立更好的基础设施。该项目将通过四个具体目标实现其目标，即扩大我们的知识和建立同行评价和遴选机制。第一个目标是通过定义能够准确描述同行评价和甄选过程特点的规范性和量化属性，为评价同行选择机制制定新的衡量标准。第二个目标是开发能够在不需要中央控制器的情况下使用的分布式对等选择机制。该项目将开发工具，以了解这些机制在这种分布式环境中的行为方式，以及为这种环境带来的独特挑战创造新机制的机会。第三个目的是加深我们对同行选择的多阶段同行评估的理解。由于许多学术会议、期刊甚至一些NSF项目的滚动审查周期，当审查(评估)可能在特定的选择设置之间传播时，需要调查同行评估和选择机制的属性。最终目的是激励同行选择的努力：在公正的经典社会选择属性之间存在根本的紧张关系，即代理人可能不会影响他们自己被接受的可能性，并为评审者在同行评价过程中投入努力提供激励。该项目将开发一个机制工具包，允许系统设计人员在代理所知的信息量、努力的激励因素和潜在的恶意行为之间理性地选择权衡。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The process of peer review, evaluation, and selection is a fundamental aspect of modern science. Funding bodies and academic publications around the world employ experts to review and select the best science for funding and publication. The process of evaluating and selecting the best from among a group of peers is much more general problem. For example, a professional society may want to give a subset of its members awards based on the opinions of all members or an instructor for a Massive Open Online Course (MOOC) may want to crowdsource grading or a marketing company may select ideas from group brainstorming sessions based on peer evaluation. In all of these settings, we wish to select a small set of winners that are judged to be the best by the community itself -- which includes those who are competing and who may have conflict of interests. This problem, known as the peer selection problem, is the focus of this research. Within a peer selection setting there may be competing priorities and inherent biases amongst the set of reviewers, and it is necessary to develop methods and algorithms that align the individual incentives of reviewers with the overall goal of selecting the best set. The intellectual merit of this project lies in expanding our understanding and developing novel algorithms for the process of peer evaluation and peer selection. Within the fields that use peer review, conflict of interest and peer selection bias have been cited as an impediment for broader participation in the science. This project will have broad impact through making the peer review process more robust to equitable selection by filtering some reviewers’ unconscious biases and conflict of interest thus resulting in a better infrastructure for research and education.The project will achieve its goal of expanding our knowledge and building mechanisms for peer evaluation and selection through four specific aims. The first aim is to develop novel metrics for the evaluation of peer selection mechanisms by defining both normative and quantitative properties that allow to precisely describe features of the peer evaluation and selection process. The second aim is to develop distributed peer selection mechanisms that are able to be used without requiring a centralized controller. This project will develop tools to understand how these mechanisms behave in this distributed setting as well as opportunities to create novel mechanisms for the unique challenges this setting poses. The third aim is to develop our understanding of multi-stage peer evaluation for peer selection. Motivated by the rolling review cycle of many academic conferences, journals, and even some NSF programs, there is a need to investigate the properties of peer evaluation and selection mechanisms when reviews (evaluations) may propagate between specific selection settings. The final aim is to incentivize effort in peer selection: There is a fundamental tension between the classic social choice properties of impartiality, i.e., an agent may not affect their own probability of getting accepted, and provide incentives for reviewers to invest effort in the peer evaluation process. This project will develop a tool kit of mechanisms that allow system designers to rationally choose tradeoffs between the amount of information an agent knows, incentives for effort, and potential for malicious behavior.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.