Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment
Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment
复制标题
DOI：
10.1609/aaai.v35i6.16611
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Ivan Stelmakh;Nihar B. Shah;Aarti Singh
中科院分区：
文献类型：
--
作者：
Ivan Stelmakh;Nihar B. Shah;Aarti Singh
我们考虑在各种同行评估任务中的战略行为问题，包括考试或家庭作业的同行评分以及在招聘或晋升时的同行审查。当同行评估任务具有竞争性时(例如，当对学生进行曲线评分时)，代理可能会受到激励，错误报告评估，以提高自己的最终排名。我们的重点是设计检测此类操纵的方法。具体地说，我们考虑了一种设置，在这种设置中，代理评估其同行的子集和输出排名，这些排名稍后被聚合以形成最终排序。在本文中，我们研究了这个问题的统计框架，并设计了一个检测战略行为的原则性检验。我们证明了我们的测试具有很强的虚警保证，并在实际环境中对其检测能力进行了评估。为此，我们设计并进行了一项实验，从受试者那里引出战略行为，并发布可能独立感兴趣的战略行为模式的数据集。我们使用这些数据进行了一系列真实和半合成的评估，表明我们的测试具有强大的检测能力。
We consider the issue of strategic behaviour in various peer-assessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions. When a peer-assessment task is competitive (e.g., when students are graded on a curve), agents may be incentivized to misreport evaluations in order to improve their own final standing. Our focus is on designing methods for detection of such manipulations. Specifically, we consider a setting in which agents evaluate a subset of their peers and output rankings that are later aggregated to form a final ordering. In this paper, we investigate a statistical framework for this problem and design a principled test for detecting strategic behaviour. We prove that our test has strong false alarm guarantees and evaluate its detection ability in practical settings. For this, we design and conduct an experiment that elicits strategic behaviour from subjects and release a dataset of patterns of strategic behaviour that may be of independent interest. We use this data to run a series of real and semi-synthetic evaluations that reveal a strong detection power of our test.