HNA: Development and evaluation of methods to assess the quality of audit data used to calculate risk-adjusted performance indicators

国家临床审计（NCAs）的一个共同目标是使用风险调整结果指标（如外科手术术后90天死亡率）检查组织的绩效，并评估组织是否达到预期的护理标准。这样的信息对相关组织很重要，因为它支持质量改进，对护理质量委员会等国家监管机构也很重要。因此，临床审计必须避免错误地将组织标记为表现不佳（假阳性）或未能检测到组织未达到预期标准（假阴性）。临床审计准确描述组织绩效的努力可能会被糟糕的数据质量所破坏。不完善的数据可能会导致结果指标值的偏差，从而导致假阳性和假阴性评估。这种后果可能会削弱专业人员和公众对审计过程的信心，以及对正在接受评估的保健服务质量的信心。最近，由于审计报告的结果不佳，一家心脏科面临关闭的威胁，这一风险得到了突出体现，但在发现医院没有提交所有病例时，这一威胁被撤销。监测和改善数据质量的策略有多种，例如检查变量值的范围，以及在发表之前与组织分享初步分析结果进行检查。尽管如此，迫切需要改进NCAs用于检查数据质量的程序。最近对审计的一项调查发现，在28次审计中，有24次审计过程的范围仅限于标记异常值，28次审计中只有14次测试了数据的可靠性。该研究的目的是开发和评估方法，以标记可能受到患者记录数据集中测量误差影响的数据点。将有两种类型的DQ旗。主标志将与变量的特定值相关，换句话说，与单个数据点相关。二级DQ标志将从数据项DQ标志衍生而来，用于描述(a)患者记录中可能出现的错误数量，以及(b)较高汇总级别（如组织级别）的数据质量评级。这些DQ标志将被设计成国家临床审计使用的共同统计程序的一部分。NCAs通常使用回归模型来产生风险调整后的结果度量，并将设计用于此过程的方法。此外，审计数据集经常包含缺失值，国家审计机构将在风险调整过程中输入缺失值。因此，DQ标志将被开发用于多重输入框架，特别是通过链式方程方法（MICE）进行多重输入。MICE方法广泛应用于临床审计，更广泛地应用于观察性研究。关于多重归责的理论和原理有大量的文献。然而，这项工作倾向于建立该方法的理论基础及其假设的有效性，当用于处理由不同机制（无论是否随机）引起的丢失数据的模式时。已经开发了一些统计工具来检查输入数据集的质量，但这些工具侧重于检查输入值的分布。在代入过程中没有标准的方法来检查数据错误，也不清楚这些数据错误如何影响代入过程的结果。拟议的研究将解决这个问题。因此，我们期望这项研究的结果对统计学家和卫生保健研究人员以及国家临床审计有价值。

A common aim of national clinical audits (NCAs) is to examine the performance of an organisation using a risk-adjusted outcome indicator, such as 90-day postoperative mortality for a surgical procedure, and assess whether the organisation is meeting expected standards of care. Such information is important both to the organisation in question, as it supports quality improvement and to national regulators like the Care Quality Commission. It is therefore essential for clinical audits to not falsely label an organisation as performing poorly (false-positive) or fail to detect when an organisation is not meeting expected standards (false-negative). The efforts of clinical audits to accurately describe organisational performance can be undermined by poor data quality. Imperfect data can introduce bias in outcome indicator values and so lead to false-positive and false-negative assessments. Such consequences can undermine the confidence of professionals and the public in the audit process as well as confidence in the quality of the health care services that are being evaluated. This risk was highlighted recently when a cardiac unit was threatened with closure due to poor outcomes reported by an audit, only for this to be retracted when it was found that the hospital had not submitted all its cases.Various strategies are available to monitor and improve data quality, such as checking the range of variable values, and sharing results of preliminary analyses with organisations for checking prior to publication. Nonetheless, there is an urgent need to improve the procedures used by NCAs to check data quality. A recent survey of Audits found that, in 24 of 28 audits, the scope of these processes is limited to flagging exceptional values, and only 14 of the 28 tested the reliability of the data.The aim of the research is to develop and evaluate methods to flag data points that are likely to be affected by measurement error within datasets of patient records. There will be two types of DQ flag. The primary flag will relate to a specific value of a variable, in other words, an individual data point. The secondary DQ flag will be derived from the data item DQ flag to describe (a) the likely number of errors within a patient record, and (b) a rating of the data quality at higher levels of aggregation, such as an organisational level.These DQ flags will be designed to become part of the common statistical procedures used by national clinical audits. NCAs typically use regression models to produce risk-adjusted outcome measures and the methods will be designed to be used during this process. In addition, audit datasets often contain missing values and NCAs will impute missing values during the risk-adjustment process. Consequently, the DQ flags will be developed to be used within a multiple imputation framework, notably, the Multiple Imputation by Chained Equations approach (MICE). The MICE approach is widely used by clinical audits and more generally, in observational research.There is an extensive literature about the theory and the principles of multiple imputation. However, this work has tended to focus on establishing the theoretical foundations of the approach and the validity of its assumptions when used to handle patterns of missing data caused by different mechanisms (whether at random or not). A few statistical tools have been developed to examine the quality of the imputed datasets but these focus on checking the distribution of imputed values. There is no standard approach to checking for data errors during the imputation process, nor a clear understanding of how these data errors might affect the results of the imputation process. The proposed research will address this issue. Consequently, we expect the results of this research to be of value to statisticians and health care researchers in general as well as to national clinical audits.