Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial.
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial.
复制标题
DOI：
10.20982/tqmp.08.1.p023
复制
发表时间：
2012
期刊：
影响因子：
--
通讯作者：
Hallgren KA
中科院分区：
文献类型：
--
作者：
Hallgren KA
许多研究设计需要评估评分者间信度（IRR），以证明多个编码器提供的观察评分之间的一致性。然而，许多研究使用了不正确的统计程序，未能充分报告解释其结果所需的信息，或者没有解决IRR如何影响其后续假设检验分析的能力。本文概述了与IRR评估相关的方法学问题，重点是研究设计，选择适当的统计数据，以及一些常用IRR统计数据的计算，解释和报告。计算示例包括SPSS和R语法，用于计算Cohen的kappa和类内相关性以评估IRR。
Many research designs require the assessment of inter-rater reliability (IRR) to demonstrate consistency among observational ratings provided by multiple coders. However, many studies use incorrect statistical procedures, fail to fully report the information necessary to interpret their results, or do not address how IRR affects the power of their subsequent analyses for hypothesis testing. This paper provides an overview of methodological issues related to the assessment of IRR with a focus on study design, selection of appropriate statistics, and the computation, interpretation, and reporting of some commonly-used IRR statistics. Computational examples include SPSS and R syntax for computing Cohen’s kappa and intra-class correlations to assess IRR.