A Bayesian Approach to Ranking and Rater Evaluation: An Application to Grant Reviews
A Bayesian Approach to Ranking and Rater Evaluation: An Application to Grant Reviews
复制标题
DOI：
10.3102/1076998609353116
复制
发表时间：
2010-04-01
影响因子：
2.4
通讯作者：
Zhang, Song
中科院分区：
文献类型：
--
作者：
Cao, Jing;Stokes, S. Lynne;Zhang, Song
我们开发了一个贝叶斯层次模型，用于分析多评价者排名研究的有序数据。评分者评分的模型包括四个潜在因素：一个是决定项目真实顺序的潜在项目特质，另外三个是评分者的绩效特征，包括评分中的偏见、歧视和测量误差。拟议的办法旨在实现三个目标。首先，三个贝叶斯估计被引入估计项目的秩。他们都表现出了显着的改善，广泛使用的评分总和，通过使用的可变技能的评分员的信息。其次，评分员的表现可以根据评分员的偏见，歧视和测量误差进行比较。第三，一个基于模拟的决策理论的方法来确定采用的评分员的数量。一个模拟研究和分析的基础上，赠款审查数据集。
We develop a Bayesian hierarchical model for the analysis of ordinal data from multirater ranking studies. The model for a rater's score includes four latent factors: one is a latent item trait determining the true order of items and the other three are the rater's performance characteristics, including bias, discrimination, and measurement error in the ratings. The proposed approach aims at three goals. First, three Bayesian estimators are introduced to estimate the ranks of items. They all show a substantial improvement over the widely used score sums by using the information on the variable skill of the raters. Second, rater performance can be compared based on rater bias, discrimination, and measurement error. Third, a simulation-based decision-theoretic approach is described to determine the number of raters to employ. A simulation study and an analysis based on a grant review data set are presented.