HNA: Development and evaluation of methods to assess the quality of audit data used to calculate risk-adjusted performance indicators

点击翻译按钮获取中文摘要

A common aim of national clinical audits (NCAs) is to examine the performance of an organisation using a risk-adjusted outcome indicator, such as 90-day postoperative mortality for a surgical procedure, and assess whether the organisation is meeting expected standards of care. Such information is important both to the organisation in question, as it supports quality improvement and to national regulators like the Care Quality Commission. It is therefore essential for clinical audits to not falsely label an organisation as performing poorly (false-positive) or fail to detect when an organisation is not meeting expected standards (false-negative). The efforts of clinical audits to accurately describe organisational performance can be undermined by poor data quality. Imperfect data can introduce bias in outcome indicator values and so lead to false-positive and false-negative assessments. Such consequences can undermine the confidence of professionals and the public in the audit process as well as confidence in the quality of the health care services that are being evaluated. This risk was highlighted recently when a cardiac unit was threatened with closure due to poor outcomes reported by an audit, only for this to be retracted when it was found that the hospital had not submitted all its cases.Various strategies are available to monitor and improve data quality, such as checking the range of variable values, and sharing results of preliminary analyses with organisations for checking prior to publication. Nonetheless, there is an urgent need to improve the procedures used by NCAs to check data quality. A recent survey of Audits found that, in 24 of 28 audits, the scope of these processes is limited to flagging exceptional values, and only 14 of the 28 tested the reliability of the data.The aim of the research is to develop and evaluate methods to flag data points that are likely to be affected by measurement error within datasets of patient records. There will be two types of DQ flag. The primary flag will relate to a specific value of a variable, in other words, an individual data point. The secondary DQ flag will be derived from the data item DQ flag to describe (a) the likely number of errors within a patient record, and (b) a rating of the data quality at higher levels of aggregation, such as an organisational level.These DQ flags will be designed to become part of the common statistical procedures used by national clinical audits. NCAs typically use regression models to produce risk-adjusted outcome measures and the methods will be designed to be used during this process. In addition, audit datasets often contain missing values and NCAs will impute missing values during the risk-adjustment process. Consequently, the DQ flags will be developed to be used within a multiple imputation framework, notably, the Multiple Imputation by Chained Equations approach (MICE). The MICE approach is widely used by clinical audits and more generally, in observational research.There is an extensive literature about the theory and the principles of multiple imputation. However, this work has tended to focus on establishing the theoretical foundations of the approach and the validity of its assumptions when used to handle patterns of missing data caused by different mechanisms (whether at random or not). A few statistical tools have been developed to examine the quality of the imputed datasets but these focus on checking the distribution of imputed values. There is no standard approach to checking for data errors during the imputation process, nor a clear understanding of how these data errors might affect the results of the imputation process. The proposed research will address this issue. Consequently, we expect the results of this research to be of value to statisticians and health care researchers in general as well as to national clinical audits.