Accounting for overlap?: An application of Mezzich's κ statistic to test interrater reliability of interview data on parental accident and emergency attendance
Accounting for overlap?: An application of Mezzich's κ statistic to test interrater reliability of interview data on parental accident and emergency attendance
复制标题
DOI：
10.1046/j.1365-2648.2001.01718.x
复制
发表时间：
2001-03-01
影响因子：
3.8
通讯作者：
MacFaul, R
中科院分区：
文献类型：
--
作者：
Eccleston, P;Werneke, U;MacFaul, R
研究理论基础。由于保健服务研究的增长，对服务使用者进行访谈研究的数量正在增加。多个访谈数据编码器之间的一致程度需要统计计算来支持结果。通常使用基本的kappa统计数据，但这是基于相互排斥的数据。研究人员应该意识到，当一个采访词或段落可以被归类为多个类别时，这是无效的。Mezzich等人（1981,Journal of psychiatry Research, 16, 29-39）提出的“比例重叠”kappa扩展作为一种原始的解决方案进行了研究。通过将Mezzich等人的“比例重叠”kappa统计应用于口头访谈数据，评估访谈数据的几个评分者之间的一致性水平。调查的临床领域是儿童在急诊科的出勤率，在那里父母的出勤经验尚未得到充分的探索。两名研究人员使用编码时间表对采访记录的随机样本进行编码。这些数据应用于Mezzich的程序；编码器1指出，一个段落指的是a类和B类，但编码器2指出的是a、B和c类。在这种情况下，协议重叠的总数为0.66，因为在三个可能的协议中达成了两个实际协议。每个段落重复这一步骤，并除以编码对的数量。所有的协议值相加，然后除以段落总数得到P-o（观察到的协议总数），除以编码对总数得到P-e（偶然达成的协议总数）。P-o和P-e在基本IC公式中用于评估访谈编码信度。总体平均P-o为0.61，平均P-e为0.32，kappa评分为0.43；中等水平的一致性，具有统计学意义（t = 4.8, P < 0.001, d.f = 23）。Mezzich的程序可以应用于访谈数据，以计算几个编码员之间的协议水平。
Study rationale. The number of interview studies with service users is rising because of growth in health services research. The level of agreement between multiple interview data coders requires statistical calculation to support results. Basic kappa statistics are often used hut this defends on having mutually exclusive data. Researchers should be aware that this is not valid when an interview word or paragraph can be coded into more than one category. The 'proportional overlap' kappa extension by Mezzich et al. (1981, Journal of Psychiatric Research 16, 29-39) has been investigated as an original solution.Objectives. To assess the level of agreement beyond chance between several raters of interview data by applying the 'proportional overlap' kappa statistic by Mezzich et al. to verbal interview data. The clinical area investigated was child attendance at an Accident and Emergency Department, where parental attendance experiences have been under-explored.Methods. Two researchers using a coding schedule coded a random sample of interview transcripts. These data were applied to Mezzich's procedure; coder 1 notes that a paragraph refers to category A and B but coder 2 notes A, B and C. The total agreement overlap in this case was 0.66 because two actual agreements out of three possible agreements were made. This was repeated fur each paragraph and divided by the number of coding pairs. All agreement values were summed then subsequently divided by the total number of paragraphs to get P-o (total number of observed agreements) and by the total number of coding pairs to get P-e (total number of agreements by chance alone). P-o and P-e were used in the basic IC formula to assess interview coding reliability.Results. The overall mean P-o was 0.61, the mean P-e was 0.32, with a kappa score of 0.43; a moderate level of agreement which was: statistically significant (t = 4.8, P < 0.001, d.f. = 23).Conclusion. Mezzich's procedure may be applied to interview data to calculate agreement levels between several coders.