PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments
PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments
复制标题
DOI：
10.1016/j.artint.2022.103843
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Omer Lev;Nicholas Mattei;P. Turrini;Stanislav Zhydkov
中科院分区：
文献类型：
--
作者：
Omer Lev;Nicholas Mattei;P. Turrini;Stanislav Zhydkov
在对等体选择中，一组代理必须选择他们自己的一个子集作为获胜者，例如，同行评审的赠款或奖金。我们采取孔多塞认为这个聚合问题，假设有一个客观的地面真理订购的代理商。我们研究的代理人对这个地面真相有一个嘈杂的感知，并给出评估，即使是真实的，也可能是不准确的。我们的目标是选择最好的一组代理，根据潜在的地面真相，通过查看潜在的不可靠的评估同行。除了可能不可靠之外，我们还允许代理人是自私的，试图影响决策的结果。因此，我们专注于解决问题ofimpartial（或strategyproof）同行选择-我们如何防止代理人操纵他们的评论，同时仍然选择最值得的个人，所有在嘈杂的评价存在？我们提出了一种新的公平的同行选择算法，PeerNomination，旨在满足上述desiderata。我们提供了一个全面的理论分析的召回对等命名和证明各种属性，包括公正性和单调性。我们还提供了基于计算机模拟的实证结果，以显示其有效性相比，国家的最先进的公平的同行选择算法。然后，我们调查的鲁棒性ofPeerNomination各种水平的噪音的评论。为了在这种情况下保持良好的性能，我们扩展PeerNomination通过使用权重的审稿人，非正式地，捕获一些概念的可靠性的审稿人。我们从理论上证明，新算法保留了策略性，从经验上讲，权重有助于识别嘈杂的评论者，从而提高选择性能。
In peer selection a group of agents must choose a subset of themselves, as winners for, e.g., peer-reviewed grants or prizes. We take a Condorcet view of this aggregation problem, assuming that there is an objective ground-truth ordering over the agents. We study agents that have a noisy perception of this ground truth and give assessments that, even when truthful, can be inaccurate. Our goal is to select the best set of agents according to the underlying ground truth by looking at the potentially unreliable assessments of the peers. Besides being potentially unreliable, we also allow agents to be self-interested, attempting to influence the outcome of the decision in their favour. Hence, we are focused on tackling the problem ofimpartial (or strategyproof) peer selection– how do we prevent agents from manipulating their reviews while still selecting the most deserving individuals, all in the presence of noisy evaluations? We propose a novel impartial peer selection algorithm,PeerNomination, that aims to fulfil the above desiderata. We provide a comprehensive theoretical analysis of the recall ofPeerNominationand prove various properties, including impartiality and monotonicity. We also provide empirical results based on computer simulations to show its effectiveness compared to the state-of-the-art impartial peer selection algorithms. We then investigate the robustness ofPeerNominationto various levels of noise in the reviews. In order to maintain good performance under such conditions, we extendPeerNominationby usingweightsfor reviewers which, informally, capture some notion of reliability of the reviewer. We show, theoretically, that the new algorithm preserves strategyproofness and, empirically, that the weights help identify the noisy reviewers and hence to increase selection performance.1