Measuring intrarater association between correlated ordinal ratings.
Measuring intrarater association between correlated ordinal ratings.
复制标题
DOI：
10.1002/bimj.201900177
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Edwards D
中科院分区：
文献类型：
--
作者：
Nelson KP;Zhou TJ;Edwards D
在影像学测试中，通常会观察到评估者的序数分数之间的差异，从而导致诊断过程的不确定性。在乳腺癌筛查中，放射科医生直观地解释乳房X光检查和核磁共振成像，而皮肤病、阿尔茨海默病和精神疾病则根据临床判断进行分级。因此，研究通常在临床环境中进行，以调查新的培训工具是否可以提高评估者的解释能力。在此类研究中，一大群专家分别在某种形式的培训之前和之后的两个不同场合对一组患者的测试结果进行分类，目的是评估培训对专家配对评分的影响。然而，由于顺序评分的相关性，很少有统计方法可以用来衡量评分者配对分数之间的关联。现有的措施仅限于在单一筛选测试的一个时间点评估关联性。我们在这里提出了一种新颖的配对 kappa，以提供评估者培训之前和之后许多评估者对患者测试结果的配对顺序评估之间的关联性的汇总测量。当评估者两次查看患者的测试结果且两次查看之间不进行干预时，评估者内部关联还可以提供有关评级一致性的宝贵见解。与现有的相关指标相比，拟议的 kappa 指标可以对两个时间点多个评估者分数之间的关联性进行总体评估，并且对潜在疾病的患病率具有稳健性。我们在最近的两项乳房成像研究中实施了我们提出的方法，并进行了广泛的模拟研究，以评估我们的关联性汇总测量的属性和性能。
Variability between raters’ ordinal scores is commonly observed in imaging tests, leading to uncertainty in the diagnostic process. In breast cancer screening, a radiologist visually interprets mammograms and MRIs, while skin diseases, Alzheimer’s disease, and psychiatric conditions are graded based on clinical judgment. Consequently, studies are often conducted in clinical settings to investigate whether a new training tool can improve the interpretive performance of raters. In such studies, a large group of experts each classify a set of patients’ test results on two separate occasions, before and after some form of training with the goal of assessing the impact of training on experts’ paired ratings. However, due to the correlated nature of the ordinal ratings, few statistical approaches are available to measure association between raters’ paired scores. Existing measures are restricted to assessing association at just one time point for a single screening test. We propose here a novel paired kappa to provide a summary measure of association between many raters’ paired ordinal assessments of patients’ test results before versus after rater training. Intrarater association also provides valuable insight into the consistency of ratings when raters view a patient’s test results on two occasions with no intervention undertaken between viewings. In contrast to existing correlated measures, the proposed kappa is a measure that provides an overall evaluation of the association among multiple raters’ scores from two time points and is robust to the underlying disease prevalence. We implement our proposed approach in two recent breast-imaging studies and conduct extensive simulation studies to evaluate properties and performance of our summary measure of association.
登录
查看更多内容
影响因子：
19.7
作者：
King, Valencia;Brooks, Jennifer D.;Morris, Elizabeth A.
通讯作者：
Morris, Elizabeth A.
影响因子：
2.7
作者：
BRENNAN, RL;PREDIGER, DJ
通讯作者：
PREDIGER, DJ
影响因子：
7.2
作者：
Hsiao, Chuhsing K.;Chen, Pei-Chun;Kao, Wen-Hsin
通讯作者：
Kao, Wen-Hsin
影响因子：
2.7
作者：
Gwet, Kilem L.
通讯作者：
Gwet, Kilem L.
DOI：
10.1001/jama.2015.1405
发表时间：
2015-03-17
期刊：
JAMA
影响因子：
--
作者：
Elmore JG;Longton GM;Carney PA;Geller BM;Onega T;Tosteson AN;Nelson HD;Pepe MS;Allison KH;Schnitt SJ;O'Malley FP;Weaver DL
通讯作者：
Weaver DL