HNA: Development and evaluation of methods to assess the quality of audit data used to calculate risk-adjusted performance indicators

国家临床审计(NCA)的一个共同目标是使用风险调整的结果指标来检查组织的业绩，例如手术后90天的死亡率，并评估组织是否达到预期的护理标准。这些信息对相关组织非常重要，因为它支持质量改进，对护理质量委员会等国家监管机构也是如此。因此，对于临床审计来说，重要的是不要错误地将组织标记为表现不佳(假阳性)或未能发现组织何时没有达到预期的标准(假阴性)。临床审计准确描述组织绩效的努力可能会因糟糕的数据质量而受到破坏。不完美的数据可能会在结果指标值中引入偏差，从而导致假阳性和假阴性评估。这种后果会破坏专业人员和公众对审计过程的信心，以及对正在评估的保健服务质量的信心。最近，当一个心脏单位由于审计报告的不良结果而面临关闭的威胁时，这种风险被突显出来，但当发现医院并未提交所有病例时，这一风险被撤回。有各种策略可用于监测和改进数据质量，例如检查变量值的范围，以及在发布之前与组织共享初步分析的结果以进行检查。尽管如此，迫切需要改进国家评估机构用来检查数据质量的程序。最近的一项审计调查发现，在28项审计中，有24项这些过程的范围仅限于标记例外值，而在28项审计中，只有14项测试了数据的可靠性。研究的目的是开发和评估在患者记录数据集中标记可能受测量误差影响的数据点的方法。将有两种类型的DQ标志。主标志将与变量的特定值相关，换句话说，是单个数据点。次要DQ标志将从数据项DQ标志中派生，以描述(A)患者记录中可能的错误数量，以及(B)较高聚合级别(如组织级别)的数据质量评级。这些DQ标志将被设计为国家临床审计使用的常见统计程序的一部分。NCAS通常使用回归模型来产生风险调整后的结果衡量标准，这些方法将在这一过程中使用。此外，审计数据集通常包含缺失值，而国家评估机构将在风险调整过程中推算缺失值。因此，DQ标志将被开发为在多重归因法框架内使用，特别是链式多重归因法(MICE)。MICE方法被广泛用于临床审计，更广泛地说，在观察性研究中。有大量关于多重归因的理论和原则的文献。然而，这项工作倾向于建立该方法的理论基础及其假设的有效性，当用于处理由不同机制(无论是随机的还是非随机的)造成的缺失数据的模式时。已经开发了一些统计工具来检查推定数据集的质量，但这些工具侧重于检查推定值的分布。没有标准的方法来检查归责过程中的数据错误，也没有清楚地了解这些数据错误可能如何影响归责过程的结果。拟议的研究将解决这一问题。因此，我们希望这项研究的结果对统计学家和卫生保健研究人员以及国家临床审计都有价值。

A common aim of national clinical audits (NCAs) is to examine the performance of an organisation using a risk-adjusted outcome indicator, such as 90-day postoperative mortality for a surgical procedure, and assess whether the organisation is meeting expected standards of care. Such information is important both to the organisation in question, as it supports quality improvement and to national regulators like the Care Quality Commission. It is therefore essential for clinical audits to not falsely label an organisation as performing poorly (false-positive) or fail to detect when an organisation is not meeting expected standards (false-negative). The efforts of clinical audits to accurately describe organisational performance can be undermined by poor data quality. Imperfect data can introduce bias in outcome indicator values and so lead to false-positive and false-negative assessments. Such consequences can undermine the confidence of professionals and the public in the audit process as well as confidence in the quality of the health care services that are being evaluated. This risk was highlighted recently when a cardiac unit was threatened with closure due to poor outcomes reported by an audit, only for this to be retracted when it was found that the hospital had not submitted all its cases.Various strategies are available to monitor and improve data quality, such as checking the range of variable values, and sharing results of preliminary analyses with organisations for checking prior to publication. Nonetheless, there is an urgent need to improve the procedures used by NCAs to check data quality. A recent survey of Audits found that, in 24 of 28 audits, the scope of these processes is limited to flagging exceptional values, and only 14 of the 28 tested the reliability of the data.The aim of the research is to develop and evaluate methods to flag data points that are likely to be affected by measurement error within datasets of patient records. There will be two types of DQ flag. The primary flag will relate to a specific value of a variable, in other words, an individual data point. The secondary DQ flag will be derived from the data item DQ flag to describe (a) the likely number of errors within a patient record, and (b) a rating of the data quality at higher levels of aggregation, such as an organisational level.These DQ flags will be designed to become part of the common statistical procedures used by national clinical audits. NCAs typically use regression models to produce risk-adjusted outcome measures and the methods will be designed to be used during this process. In addition, audit datasets often contain missing values and NCAs will impute missing values during the risk-adjustment process. Consequently, the DQ flags will be developed to be used within a multiple imputation framework, notably, the Multiple Imputation by Chained Equations approach (MICE). The MICE approach is widely used by clinical audits and more generally, in observational research.There is an extensive literature about the theory and the principles of multiple imputation. However, this work has tended to focus on establishing the theoretical foundations of the approach and the validity of its assumptions when used to handle patterns of missing data caused by different mechanisms (whether at random or not). A few statistical tools have been developed to examine the quality of the imputed datasets but these focus on checking the distribution of imputed values. There is no standard approach to checking for data errors during the imputation process, nor a clear understanding of how these data errors might affect the results of the imputation process. The proposed research will address this issue. Consequently, we expect the results of this research to be of value to statisticians and health care researchers in general as well as to national clinical audits.