RI: Small: Robustness to Undesirable Behavior in Peer Review

RI：小：同行评审中对不良行为的鲁棒性

• 批准号: 2200410
• 负责人: Nihar Shah
• 金额: $ 60万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-06-15 至 2025-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2200410&HistoricalAwards=false
• 关键词: RI; Small; Robustness; Undesirable; Behavior

Peer review is the backbone of scientific research. It is used widely to review the quality of papers as well as for awarding grants worth billions of dollars every year. However, due to more quantitative evaluations for jobs and promotions, incentives to have papers or proposals accepted are only growing. While the vast majority of the research community consists of honest people, recently there have been discoveries of a certain kind of fraud in peer review. In this type of fraud, a group of authors and reviewers form a coalition outside of the review process, and aim to get assigned each others' papers or proposals and then illegitimately accept them. If left untreated, there is significant risk of bad behavior spreading. The presence and prevalence of such malicious activities also leads to increased skepticism and distrust in science among the general public. The proposed research will have a broad impact in terms of addressing these critical issues. This research will develop a computational toolkit for organizers of peer review to detect and/or mitigate such fraud. The researchers will also assist any organizers who wish to employ this toolkit.The large number of submissions in many conferences and the increase in scale and sophistication of manipulation attempts makes it infeasible to manually detect and/or mitigate these issues. This research will design a theoretical framework that explicitly considers coalition-based fraud, and allows to make precise claims about guarantees that the algorithms give us. The research will also design algorithms that achieve desirable guarantees within that framework. These will be built into an open-sourced toolkit that will be made publicly available for use. The research will design and use techniques spanning machine learning, optimization, statistics, and game theory and mechanism design. The research will advance the understanding of the fundamental limits of mitigating collusions in various peer-assessment settings, and how these limits can be achieved or approached through the design of scalable and practical algorithms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

同行评议是科学研究的支柱。它被广泛用于审查论文的质量，以及每年授予价值数十亿美元的拨款。然而，由于对工作和晋升进行了更多的量化评估，接受论文或提案的动机只会越来越大。虽然研究界的绝大多数人都是诚实的人，但最近发现了同行评议中的某种欺诈行为。在这种类型的欺诈中，一群作者和审稿人在审查过程之外组成联盟，目的是获得彼此的论文或提案，然后非法接受它们。如果不治疗，不良行为传播的风险很大。这种恶意活动的存在和流行也导致公众对科学的怀疑和不信任增加。拟议的研究将在解决这些关键问题方面产生广泛影响。这项研究将为同行审查组织者开发一个计算工具包，以发现和/或减轻此类欺诈行为。研究人员还将帮助任何希望使用该工具包的组织者。许多会议上的大量提交以及操纵尝试的规模和复杂性的增加使得手动检测和/或缓解这些问题是不可行的。这项研究将设计一个明确考虑基于联盟的欺诈的理论框架，并允许对算法给我们的保证做出准确的声明。这项研究还将设计在该框架内实现理想保证的算法。这些将被构建成一个开放源码的工具包，并将公开提供使用。这项研究将设计和使用跨越机器学习、优化、统计以及博弈论和机制设计的技术。这项研究将促进对在各种同行评估环境中缓解串通的基本限制的理解，以及如何通过设计可扩展和实用的算法来达到或接近这些限制。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。