Loss Functions, Axioms, and Peer Review
Loss Functions, Axioms, and Peer Review
复制标题
DOI：
10.1613/jair.1.12554
复制
发表时间：
2018-08
期刊：
影响因子：
--
通讯作者：
Ritesh Noothigattu;Nihar B. Shah;Ariel D. Procaccia
中科院分区：
文献类型：
--
作者：
Ritesh Noothigattu;Nihar B. Shah;Ariel D. Procaccia
少数评论者拒绝一篇非常新奇的论文是很常见的，因为他们认为，比方说，广泛的实验远比新颖性重要得多，而整个社区都会接受这篇论文。更广泛地说，不同审评人的标准分数与最终建议之间的不同对应是同行审查中不一致的一个主要来源。在这篇文章中，我们提出了一个受经验风险最小化(ERM)启发的框架，用于学习社区的聚合映射。出现的关键挑战是确定ERM的损失函数。我们考虑L(p，q)损失函数类，它是标准的Lp损失类在向量上的矩阵推广；这里损失函数的选择相当于选择超参数p和q。为了处理我们问题中基本事实的缺乏，我们利用计算社会选择来确定超参数p和q的合意值。具体地说，我们刻画了p=q=1是这些超参数中满足三个自然公理性质的唯一选择。最后，我们将我们的方法实施并应用于IJCA2017年的审查。
It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper. More generally, the disparate mapping of criteria scores to final recommendations by different reviewers is a major source of inconsistency in peer review. In this paper we present a framework inspired by empirical risk minimization (ERM) for learning the community's aggregate mapping. The key challenge that arises is the specification of a loss function for ERM. We consider the class of L(p,q) loss functions, which is a matrix-extension of the standard class of Lp losses on vectors; here the choice of the loss function amounts to choosing the hyperparameters p and q. To deal with the absence of ground truth in our problem, we instead draw on computational social choice to identify desirable values of the hyperparameters p and q. Specifically, we characterize p=q=1 as the only choice of these hyperparameters that satisfies three natural axiomatic properties. Finally, we implement and apply our approach to reviews from IJCAI 2017.