PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review
PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review
复制标题
DOI：
--
复制
发表时间：
2018-06
期刊：
影响因子：
--
通讯作者：
Ivan Stelmakh;Nihar B. Shah;Aarti Singh
中科院分区：
文献类型：
--
作者：
Ivan Stelmakh;Nihar B. Shah;Aarti Singh
我们考虑在会议同行评审中自动将论文分配给审稿人的问题，重点关注公平性和统计准确性。我们的公平目标是最大化最弱势论文的审稿质量，而不是最大化所有论文的总体质量的常用目标。我们设计了一种基于增量最大流程序的分配算法，我们证明该算法是近乎最优公平的。我们的统计准确性目标是确保正确恢复应被接受的论文。我们为流行的客观评分模型以及我们在论文中提出的新颖的主观评分模型的同行评审过程的准确性提供了尖锐的极小极大分析。我们的分析证明，我们提出的分配算法也能带来接近最佳的统计准确性。最后，我们设计了一个新颖的实验，可以对各种分配算法进行客观比较，并克服同行评审实验中缺乏基本事实所带来的固有困难。该实验以及对合成数据和真实数据的其他实验的结果证实了我们算法的理论保证。
We consider the problem of automated assignment of papers to reviewers in conference peer review, with a focus on fairness and statistical accuracy. Our fairness objective is to maximize the review quality of the most disadvantaged paper, in contrast to the commonly used objective of maximizing the total quality over all papers. We design an assignment algorithm based on an incremental max-flow procedure that we prove is near-optimally fair. Our statistical accuracy objective is to ensure correct recovery of the papers that should be accepted. We provide a sharp minimax analysis of the accuracy of the peer-review process for a popular objective-score model as well as for a novel subjective-score model that we propose in the paper. Our analysis proves that our proposed assignment algorithm also leads to a near-optimal statistical accuracy. Finally, we design a novel experiment that allows for an objective comparison of various assignment algorithms, and overcomes the inherent difficulty posed by the absence of a ground truth in experiments on peer-review. The results of this experiment as well as of other experiments on synthetic and real data corroborate the theoretical guarantees of our algorithm.