RI: Small: Robustness to Undesirable Behavior in Peer Review

同行评议是科学研究的支柱。它被广泛用于审查论文的质量以及每年价值数十亿美元的拨款。然而，由于对工作和晋升的量化评估越来越多，论文或提案被接受的激励措施只会越来越多。虽然绝大多数研究界的人都是诚实的，但最近在同行评审中发现了某种欺诈行为。在这种类型的欺诈中，一群作者和审稿人在审稿过程之外结成联盟，目的是获得对方的论文或提案，然后非法接受它们。如果不及时治疗，不良行为传播的风险很大。这种恶意活动的存在和流行也导致公众对科学的怀疑和不信任增加。拟议的研究将在解决这些关键问题方面产生广泛的影响。这项研究将为同行评审组织者开发一个计算工具包，以检测和/或减轻此类欺诈行为。研究人员还将协助任何希望使用此工具包的组织者。许多会议上的大量提交以及操纵尝试的规模和复杂性的增加使得手工检测和/或减轻这些问题变得不可行。这项研究将设计一个理论框架，明确考虑基于联盟的欺诈行为，并允许对算法给我们的保证做出精确的声明。该研究还将设计算法，在该框架内实现理想的保证。这些将被构建到一个开放源代码的工具包中，供公众使用。这项研究将设计和使用机器学习、优化、统计学、博弈论和机制设计等技术。该研究将促进对各种同行评估设置中减轻合谋的基本限制的理解，以及如何通过设计可扩展和实用的算法来实现或接近这些限制。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Peer review is the backbone of scientific research. It is used widely to review the quality of papers as well as for awarding grants worth billions of dollars every year. However, due to more quantitative evaluations for jobs and promotions, incentives to have papers or proposals accepted are only growing. While the vast majority of the research community consists of honest people, recently there have been discoveries of a certain kind of fraud in peer review. In this type of fraud, a group of authors and reviewers form a coalition outside of the review process, and aim to get assigned each others' papers or proposals and then illegitimately accept them. If left untreated, there is significant risk of bad behavior spreading. The presence and prevalence of such malicious activities also leads to increased skepticism and distrust in science among the general public. The proposed research will have a broad impact in terms of addressing these critical issues. This research will develop a computational toolkit for organizers of peer review to detect and/or mitigate such fraud. The researchers will also assist any organizers who wish to employ this toolkit.The large number of submissions in many conferences and the increase in scale and sophistication of manipulation attempts makes it infeasible to manually detect and/or mitigate these issues. This research will design a theoretical framework that explicitly considers coalition-based fraud, and allows to make precise claims about guarantees that the algorithms give us. The research will also design algorithms that achieve desirable guarantees within that framework. These will be built into an open-sourced toolkit that will be made publicly available for use. The research will design and use techniques spanning machine learning, optimization, statistics, and game theory and mechanism design. The research will advance the understanding of the fundamental limits of mitigating collusions in various peer-assessment settings, and how these limits can be achieved or approached through the design of scalable and practical algorithms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.