Toward Robust Information: Data Quality and Inter-Rater Reliability in the American College of Surgeons National Surgical Quality Improvement Program
Toward Robust Information: Data Quality and Inter-Rater Reliability in the American College of Surgeons National Surgical Quality Improvement Program
复制标题
DOI：
10.1016/j.jamcollsurg.2009.09.031
复制
发表时间：
2010-01-01
影响因子：
5.2
通讯作者：
Hall, Bruce L.
中科院分区：
文献类型：
--
作者：
Shiloach, Mira;Frencher, Stanley K., Jr.;Hall, Bruce L.
背景：用于评估医疗质量的数据需要具有高可靠性，以确保有效的质量评估和基准。美国外科医师学会国家外科质量改进计划（ACS NSQIP）不断强调通过其程序infrastructure.Study设计收集高度可靠的临床数据：我们提供了一个详细的描述，在ACS NSQIP使用的各种机制，以确保收集高质量的数据，包括数据收集者（外科临床审查员）的培训和持续的审计数据的可靠性。对于2005年至2008年的日历年，使用数据收集者和审计员之间的一致性百分比计算总体和单个变量的评分者间可靠性。具有> 5%不一致性的变量被标记用于教育工作以改进准确收集。科恩的kappa估计选定的变量从2007年的审计year.RESULTS：评分员间的可靠性审计表明，变量的总体分歧率已从2005年的3.15%（ACS NSQIP的第一年公开招生）下降到2008年的1.56%。此外，个别变量的分歧程度不断提高，2005年有26个个别变量的分歧超过5%，2008年只有2个。估计kappa值建议大量或几乎完美的协议，为大多数variables.CONCLUSIONS：ACS NSQIP已实施培训和审计程序，其医院参与者是非常有效的收集强大的数据。审计结果表明，自该方案开始以来，数据一直是可靠的，而且可靠性每年都在提高。（美国科尔外科杂志2010;210：6-16。(C)2010年美国外科医生学会）
BACKGROUND: Data used for evaluating quality of medical care need to be of high reliability to ensure valid quality assessment and benchmarking. The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) has continually emphasized the collection of highly reliable clinical data through its program infrastructure.STUDY DESIGN: We provide a detailed description of the various mechanisms used in ACS NSQIP to assure collection of high quality data, including training of data collectors (surgical clinical reviewers) and ongoing audits of data reliability. For the 2005 through 2008 calendar years, inter-rater reliability was calculated overall and for individual variables using percentages of agreement between the data collector and the auditor. Variables with > 5% disagreement are flagged for educational efforts to improve accurate collection. Cohen's kappa was estimated for selected variables from the 2007 audit year.RESULTS: Inter-rater reliability audits show that overall disagreement rates on variables have fallen from 3.15% in 2005 (the first year of public enrollment in ACS NSQIP) to 1.56% in 2008. In addition, disagreement levels for individual variables have continually improved, with 26 individual variables demonstrating > 5% disagreement in 2005, to only 2 such variables in 2008. Estimated kappa values suggest substantial or almost perfect agreement for most variables.CONCLUSIONS: The ACS NSQIP has implemented training and audit procedures for its hospital participants that are highly effective in collecting robust data. Audit results show that data have been reliable since the program's inception and that reliability has improved every year. (J Am Coll Surg 2010;210:6-16. (C) 2010 by the American College of Surgeons)