Refinement: Measuring informativeness of ratings in the absence of a gold standard
Refinement: Measuring informativeness of ratings in the absence of a gold standard
复制标题
DOI：
10.1111/bmsp.12268
复制
发表时间：
2022-03-16
影响因子：
2.6
通讯作者：
Lee, Carole
中科院分区：
文献类型：
--
作者：
Grant, Sheridan;Meila, Marina;Lee, Carole
我们提出了一种新的度量标准，用于评估给定尺度上单个评分者的一组评分的信息性。当评分者在同一尺度上对许多可比较的项目进行评分时，就像在招聘、大学录取和同行评议中发生的那样，这种评估是有意义的。我们的论述以同行评议为背景，其中涉及单变量和多变量基数评级。我们利用这一背景来激发一组评级的改进的信息理论措施-熵改进-以及两个次要措施。对这三个度量的数学分析表明，只有第一个度量（捕获评级的信息内容）具有适合于细化度量的属性。最后，我们分析了现实世界拨款审查数据的细化，发现总体绩效分数比标准分数更细化的证据。
We propose a new metric for evaluating the informativeness of a set of ratings from a single rater on a given scale. Such evaluations are of interest when raters rate numerous comparable items on the same scale, as occurs in hiring, college admissions, and peer review. Our exposition takes the context of peer review, which involves univariate and multivariate cardinal ratings. We draw on this context to motivate an information-theoretic measure of the refinement of a set of ratings - entropic refinement - as well as two secondary measures. A mathematical analysis of the three measures reveals that only the first, which captures the information content of the ratings, possesses properties appropriate to a refinement metric. Finally, we analyse refinement in real-world grant-review data, finding evidence that overall merit scores are more refined than criterion scores.