Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments
Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Steven Jecmen;Hanrui Zhang;Ryan Liu;Nihar B. Shah;Vincent Conitzer;Fei Fang
中科院分区：
文献类型：
--
作者：
Steven Jecmen;Hanrui Zhang;Ryan Liu;Nihar B. Shah;Vincent Conitzer;Fei Fang
我们考虑了会议同行评审中的三个重要挑战：（i）评审员恶意地试图被分配到某些论文以提供积极的评论，可能是与作者的交换协议的一部分;（ii）“鱼雷评审”，评审员故意试图被分配到他们不喜欢的某些论文以拒绝它们;（iii）在发布相似性和评审员分配代码时进行评审员去匿名化。在概念方面，我们确定了这三个问题之间的联系，并提出了一个框架，将所有这些挑战放在一个共同的保护伞下。然后，我们提出了一个（随机）算法的审稿人分配，可以最佳地解决审稿人分配问题的概率分配任何审稿人对任何给定的约束下。我们进一步考虑的问题，限制联合概率，某些可疑对的审稿人被分配到某些文件，并表明，这个问题是NP-困难的任意约束这些联合概率，但有效地解决了一个实际的特殊情况。最后，我们在过去会议的数据集上对我们的算法进行了实验评估，我们观察到它们可以将任何恶意审稿人被分配到他们想要的论文的机会限制在50%，同时产生超过90%的总最佳相似度的分配。我们的算法仍然实现了这种相似性，同时也防止了具有密切关联的审稿人被分配到同一篇论文。
We consider three important challenges in conference peer review: (i) reviewers maliciously attempting to get assigned to certain papers to provide positive reviews, possibly as part of quid-pro-quo arrangements with the authors; (ii) "torpedo reviewing," where reviewers deliberately attempt to get assigned to certain papers that they dislike in order to reject them; (iii) reviewer de-anonymization on release of the similarities and the reviewer-assignment code. On the conceptual front, we identify connections between these three problems and present a framework that brings all these challenges under a common umbrella. We then present a (randomized) algorithm for reviewer assignment that can optimally solve the reviewer-assignment problem under any given constraints on the probability of assignment for any reviewer-paper pair. We further consider the problem of restricting the joint probability that certain suspect pairs of reviewers are assigned to certain papers, and show that this problem is NP-hard for arbitrary constraints on these joint probabilities but efficiently solvable for a practical special case. Finally, we experimentally evaluate our algorithms on datasets from past conferences, where we observe that they can limit the chance that any malicious reviewer gets assigned to their desired paper to 50% while producing assignments with over 90% of the total optimal similarity. Our algorithms still achieve this similarity while also preventing reviewers with close associations from being assigned to the same paper.