Interrater Agreement Statistics With Skewed Data: Evaluation of Alternatives to Cohen's Kappa
Interrater Agreement Statistics With Skewed Data: Evaluation of Alternatives to Cohen's Kappa
复制标题
DOI：
10.1037/a0037489
复制
发表时间：
2014-12-01
影响因子：
5.9
通讯作者：
Lorber, Michael F.
中科院分区：
文献类型：
--
作者：
Xu, Shu;Lorber, Michael F.
Objective: In this study, we aimed to evaluate interrater agreement statistics (IRAS) for use in research on low base rate clinical diagnoses or observed behaviors. Establishing and reporting sufficient interrater agreement is essential in such studies. Yet the most commonly applied agreement statistic, Cohen's kappa, has a well known sensitivity to base rates that results in a substantial penalization of interrater agreement when behaviors or diagnoses are very uncommon, a prevalent and frustrating concern in such studies. Method: We performed Monte Carlo simulations to evaluate the performance of 5 of kappa's alternatives (Van Eerdewegh's V, Yule's Y, Holley and Guilford's G, Scott's pi, and Gwet's AC(1)), alongside kappa itself. The simulations investigated the robustness of these IRAS to conditions that are common in clinical research, with varying levels of behavior or diagnosis base rate, rater bias, observed interrater agreement, and sample size. Results: When the base rate was 0.5, each IRAS provided similar estimates, particularly with unbiased raters. G was the least sensitive of the IRAS to base rates. Conclusions: The results encourage the use of the G statistic for its consistent performance across the simulation conditions. We recommend separately reporting the rates of agreement on the presence and absence of a behavior or diagnosis alongside G as an index of chance corrected overall agreement.
