Debiasing Evaluations That are Biased by Evaluations
Debiasing Evaluations That are Biased by Evaluations
复制标题
DOI：
10.1609/aaai.v35i11.17214
复制
发表时间：
2020-12
期刊：
影响因子：
--
通讯作者：
Jingyan Wang;Ivan Stelmakh;Yuting Wei
中科院分区：
文献类型：
--
作者：
Jingyan Wang;Ivan Stelmakh;Yuting Wei
通过让人们给一组项目打分来评估它们是很常见的。例如，大学要求学生评价教师的教学质量，会议组织者要求投稿作者评估评论的质量。然而，在这些申请中，如果学生在一门课程中获得更高的分数，他们通常会给这门课程更高的分数，如果作者的论文被会议接受，他们通常会给评论更高的分数。在这项工作中，我们将这些外部因素称为人们所经历的“结果”，并考虑当有关结果的一些信息可用时，在给定评级中减轻这些结果引起的偏差的问题。我们将关于结果的信息表述为偏差上已知的偏序。我们提出了一种通过求解该排序约束下的正则化优化问题来消除偏差的方法，并提供了一种精心设计的自适应选择适当正则化量的交叉验证方法。我们对算法的性能提供理论保证，以及实验评估。
It is common to evaluate a set of items by soliciting people to rate them. For example, universities ask students to rate the teaching quality of their instructors, and conference organizers ask authors of submissions to evaluate the quality of the reviews. However, in these applications, students often give a higher rating to a course if they receive higher grades in a course, and authors often give a higher rating to the reviews if their papers are accepted to the conference. In this work, we call these external factors the "outcome" experienced by people, and consider the problem of mitigating these outcome-induced biases in the given ratings when some information about the outcome is available. We formulate the information about the outcome as a known partial ordering on the bias. We propose a debiasing method by solving a regularized optimization problem under this ordering constraint, and also provide a carefully designed cross-validation method that adaptively chooses the appropriate amount of regularization. We provide theoretical guarantees on the performance of our algorithm, as well as experimental evaluations.