A SUPER* Algorithm to Optimize Paper Bidding in Peer Review
A SUPER* Algorithm to Optimize Paper Bidding in Peer Review
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Tanner Fiez;Nihar B. Shah;L. Ratliff
中科院分区：
文献类型：
--
作者：
Tanner Fiez;Nihar B. Shah;L. Ratliff
许多应用程序涉及用户的顺序到达，并需要向每个用户显示商品的订购。一个最好的例子(这是本文的重点)是会议同行评议中的投标过程，评审员按顺序进入系统，需要向每位评审员展示提交的论文列表，然后评审员“出价”审查一些论文。由于优先效应，所显示的文件顺序对投标有重大影响。在决定展示论文的顺序时，有两个相互竞争的目标：(1)为每篇论文争取足够多的出价；(2)通过展示相关项目来满足审查者的要求。在本文中，我们首先开发一个框架，以原则性的方式研究这个问题。受A*算法的启发，我们提出了一种称为Super*的算法来实现这一目标。理论上，我们证明了算法的局部最优性保证，并证明了流行的基线是相当次优的。此外，在针对相似性的社区模型下，我们证明了Super*是接近最优的，而流行的基线是相当次优的。在对ICLR 2018年的真实数据和合成数据进行的实验中，我们发现Super*的性能大大优于现有系统中部署的基线，将低于必需出价的论文数量持续减少了50%-75%或更多，并且对各种现实世界的复杂性也很健壮。
A number of applications involve sequential arrival of users, and require showing each user an ordering of items. A prime example (which forms the focus of this paper) is the bidding process in conference peer review where reviewers enter the system sequentially, each reviewer needs to be shown the list of submitted papers, and the reviewer then "bids" to review some papers. The order of the papers shown has a significant impact on the bids due to primacy effects. In deciding on the ordering of papers to show, there are two competing goals: (i) obtaining sufficiently many bids for each paper, and (ii) satisfying reviewers by showing them relevant items. In this paper, we begin by developing a framework to study this problem in a principled manner. We present an algorithm called SUPER*, inspired by the A* algorithm, for this goal. Theoretically, we show a local optimality guarantee of our algorithm and prove that popular baselines are considerably suboptimal. Moreover, under a community model for the similarities, we prove that SUPER* is near-optimal whereas the popular baselines are considerably suboptimal. In experiments on real data from ICLR 2018 and synthetic data, we find that SUPER* considerably outperforms baselines deployed in existing systems, consistently reducing the number of papers with fewer than requisite bids by 50-75% or more, and is also robust to various real world complexities.