Inter-rater and test-retest reliability of quality assessments by novice student raters using the Jadad and Newcastle-Ottawa Scales.
Inter-rater and test-retest reliability of quality assessments by novice student raters using the Jadad and Newcastle-Ottawa Scales.
复制标题
DOI：
10.1136/bmjopen-2012-001368
复制
发表时间：
2012
期刊：
影响因子：
2.9
通讯作者：
ECT & Cognition Systematic Review Team
中科院分区：
文献类型：
--
作者：
Oremus M;Oremus C;Hall GB;McKinnon MC;ECT & Cognition Systematic Review Team
纳入研究的质量评估是系统评价的重要组成部分。作者调查了由缺乏经验的学生评分员进行的质量评估的评分员间和重测信度。学生评分员接受了关于质量评估的培训课程，使用Jadad量表进行随机对照试验，纽卡斯尔-渥太华量表(NOS)用于观察性研究。评分员被随机分配到五对，他们每人独立地对13-20篇文章的质量进行评分。这些文章摘自78篇研究电休克治疗严重抑郁障碍后认知障碍的论文。文章被随机分发给评分者。两个月后，每个评分者重新评估了他们分配的一半文章的质量。麦克马斯特综合神经科学发现和研究计划。10名学生参加麦克马斯特综合神经科学发现和学习计划课程。作者使用κ和组内相关系数2，1或ICC(2，1)来衡量评分者之间的可靠性。作者使用ICC(2，1)来测量重测信度。评分员之间的信度因量表问题而异。在6个条目的Jadad量表中，问题特异性κS的范围从0.13(95%CI−0.11至0.37)到0.56(95%CI 0.29至0.83)。病例对照的−为0.14(95%CI−0.28~0.00)至0.39(95%CI−0.02~0.81)，−0.20(95%CI−0.49~0.09)~1.00(95%CI 1.00~1.00)。在6个条目的Jadad量表上，ICC(2，1)S的评分员间信度和重测信度(考虑评分者之间的系统差异)分别为0.32(95%CI 0.08~0.52)和0.55(95%CI 0.41~0.67)。病例对照组的−(2，1)S分别为0.46(95%CI−0.13~0.92)和0.83(95%CI−0.48~0.95)。评分员之间的信度一般为差到一般，重测信度为一般到优。在评分员培训之后进行试点评级阶段可能是提高一致性的一种方式。对系统评价中缺乏经验的评分员对文章进行质量评估，以检验评价员之间和重测之间的可靠性。在缺乏经验的评分员中，使用Jadad量表和纽卡斯尔-渥太华量表的评分员之间的信度一般为差到一般；重测信度为一般到优秀。系统评价员必须特别注意培训缺乏经验的质量评分员；试点评分阶段可能是提高缺乏经验的评分员可靠性的有用手段，特别是在对观察性研究质量进行评分时。没有其他研究对一组缺乏经验的评分者进行质量评估的可靠性检验。结果可能会因评分员的背景和经验、评分员培训、质量评估工具和正在研究的主题而有所不同。
Quality assessment of included studies is an important component of systematic reviews. The authors investigated inter-rater and test–retest reliability for quality assessments conducted by inexperienced student raters. Student raters received a training session on quality assessment using the Jadad Scale for randomised controlled trials and the Newcastle–Ottawa Scale (NOS) for observational studies. Raters were randomly assigned into five pairs and they each independently rated the quality of 13–20 articles. These articles were drawn from a pool of 78 papers examining cognitive impairment following electroconvulsive therapy to treat major depressive disorder. The articles were randomly distributed to the raters. Two months later, each rater re-assessed the quality of half of their assigned articles. McMaster Integrative Neuroscience Discovery and Study Program. 10 students taking McMaster Integrative Neuroscience Discovery and Study Program courses. The authors measured inter-rater reliability using κ and the intraclass correlation coefficient type 2,1 or ICC(2,1). The authors measured test–retest reliability using ICC(2,1). Inter-rater reliability varied by scale question. For the six-item Jadad Scale, question-specific κs ranged from 0.13 (95% CI −0.11 to 0.37) to 0.56 (95% CI 0.29 to 0.83). The ranges were −0.14 (95% CI −0.28 to 0.00) to 0.39 (95% CI −0.02 to 0.81) for the NOS cohort and −0.20 (95% CI −0.49 to 0.09) to 1.00 (95% CI 1.00 to 1.00) for the NOS case–control. For overall scores on the six-item Jadad Scale, ICC(2,1)s for inter-rater and test–retest reliability (accounting for systematic differences between raters) were 0.32 (95% CI 0.08 to 0.52) and 0.55 (95% CI 0.41 to 0.67), respectively. Corresponding ICC(2,1)s for the NOS cohort were −0.19 (95% CI −0.67 to 0.35) and 0.62 (95% CI 0.25 to 0.83), and for the NOS case–control, the ICC(2,1)s were 0.46 (95% CI −0.13 to 0.92) and 0.83 (95% CI 0.48 to 0.95). Inter-rater reliability was generally poor to fair and test–retest reliability was fair to excellent. A pilot rating phase following rater training may be one way to improve agreement. To examine the inter-rater and test–retest reliability of inexperienced raters' quality assessments of articles included in a systematic review. Among inexperienced raters, inter-rater reliability using the Jadad Scale and Newcastle–Ottawa Scale was generally poor to fair; test–retest reliability was fair to excellent. Systematic reviewers must pay special attention to training inexperienced quality raters; a pilot rating phase might be a helpful means of improving reliability among inexperienced raters, especially when rating observational study quality. No other study has examined the reliability of quality assessments in a group of inexperienced raters. Results may differ depending on rater background and experience, rater training, quality assessment instruments and topic under study.
登录
查看更多内容
DOI：
10.1136/bmj.b2535
发表时间：
2009-07-21
期刊：
BMJ (Clinical research ed.)
影响因子：
--
作者：
Moher D;Liberati A;Tetzlaff J;Altman DG;PRISMA Group
通讯作者：
PRISMA Group
影响因子：
7.7
作者：
Sanderson, Simon;Tatt, Lain D.;Higgins, Julian P. T.
通讯作者：
Higgins, Julian P. T.
影响因子：
3.2
作者：
Maher, CG;Sherrington, C;Elkins, M
通讯作者：
Elkins, M
影响因子：
6.3
作者：
Downs, SH;Black, N
通讯作者：
Black, N
DOI：
10.1111/j.1365-2753.2003.00411.x
发表时间：
2004-05-01
影响因子：
2.4
作者：
Haywood, KL;Hargreaves, J;Lamb, SE
通讯作者：
Lamb, SE