Measuring and Understanding Diagnostic Quality from Large-Scale Data

几十年来，诊断错误一直是提高医疗保健质量努力的盲点。 与可用于评估治疗质量的多种指标相比，临床医生和政策制定者 几乎没有工具可以用来衡量和提高诊断决策的质量。没有更好的方法 要在临床医生层面系统地衡量诊断决策的质量，仍将很难 识别诊断错误的模式，按规模对类型和原因进行分类，并制定和评估 预防它们的干预措施。我们的长期目标是开发工具来衡量临床诊断质量 来自大规模数据的供应商，并建立框架和知识以将这些措施转化为 适当的干预措施。本应用程序的目标是应用和验证一种测量系统 550万人次胸部就诊中肺炎诊断中放射科医生的诊断质量 退伍军人健康管理局(VHA)急诊科(ED)的X光检查。在这个项目中，我们将解决 衡量诊断护理质量的任何数据驱动的方法都面临三个基本挑战。第一个是一个 缺乏可以作为诊断基准的可观察的基本事实，特别是在大规模数据中。这 当策略试图平衡I类错误(误报)和II类错误时，挑战尤其成问题 错误(假阴性)。其次，诊断错误率取决于潜在的疾病流行率 患者人数，这可能是不完全观察。第三，每个临床医生的小病例数可以 临床医生之间的比较复杂，因为测量的差异可能反映潜在的诊断 质量或可能由随机噪声引起。我们将通过一种新的方法组合来应对这些挑战 来自统计分类和应用经济学，建立在先前工作的基础上。我们提出了以下具体建议 目标：(1)我们将验证肺炎诊断和诊断结果的数据驱动措施。在之前 概念性工作建立在计量经济学文献选择的基础上，我们表明我们可以推断出相对 诊断质量的差异-作为类型I错误率和类型II错误率的差异-即使是单独的 在将病例分配给放射科医生的准实验分配下，第一类错误是不可观察到的；(2)我们将 在接收-操作曲线(ROC)框架中解释第一类错误和第二类错误的提供者级别比率 哪些诊断错误可能来自不正确的诊断阈值(权衡类型I和类型II错误)或 诊断准确性差(导致过多的类型I错误和类型II错误)；以及(3)探索 临床医生诊断质量的决定因素，我们将把我们的放射科医生诊断质量的测量与 他们的特征和行动跨越了数千名放射科医生。为了评估潜在的后果，我们将 研究被分配给不同诊断质量的放射科医生的准试验性患者的健康结果。 我们的项目将为临床提供者的诊断质量数据驱动测量奠定基础， 这是了解和改进我们医疗保健系统诊断性能的必要的第一步。

For decades, diagnostic errors have constituted a blind spot in the effort to improve health care quality. Compared with the multitude of metrics available to assess the quality of treatment, clinicians and policymakers have few tools with which to measure and improve the quality of diagnostic decisions. Without better methods to systematically measure the quality of diagnostic decisions at the clinician level, it will continue to be difficult to identify patterns in diagnostic errors, categorize types and causes at scale, and develop and evaluate interventions to prevent them. Our long-term goal is to develop tools to measure diagnostic quality across clinical providers from large-scale data, and to build frameworks and knowledge to translate those measures into appropriate interventions. The objective of this application is to apply and validate a system for measuring diagnostic quality across radiologists in the setting of pneumonia diagnosis among 5.5 million visits with chest X-rays in Veterans Health Administration (VHA) emergency departments (EDs). In this project, we will address three challenges fundamental to any data-driven approach to measuring quality of diagnostic care. The first is a lack of observable ground truth against which to benchmark diagnoses, particularly in large-scale data. This challenge is particularly problematic when policies seek to balance type I errors (false positives) against type II errors (false negatives). Second, rates of diagnostic errors depend on the underlying prevalence of disease in the patient population, which may be incompletely observed. Third, small case numbers per clinician can complicate comparisons between clinicians, since measured differences may reflect underlying diagnostic quality or may arise from random noise. We will address these challenges with a novel combination of methods from statistical classification and applied economics, building on prior work. We propose the following specific aims: (1) We will validate data-driven measures of pneumonia diagnoses and diagnostic outcomes. In prior conceptual work building on the econometric literature of selection, we show that we may infer relative differences in diagnostic quality—as differences in type I error rates and type II error rates—even if individual type I errors are unobservable, under quasi-experimental assignment of cases to radiologists; (2) We will interpret provider-level rates of type I error and type II error in a receiver-operating curve (ROC) framework in which diagnostic errors may arise from incorrect diagnostic thresholds (trading off type I and type II errors) or poor diagnostic accuracy (incurring both too many type I errors and type II errors); and (3) To explore the determinants of clinician diagnostic quality, we will correlate our measures of radiologist diagnostic quality with their characteristics and actions across thousands of radiologists. To assess the potential consequences, we will study health outcomes of patients quasi-experimentally assigned to radiologists of differing diagnostic quality. Our project will lay the groundwork for data-driven measurement of diagnostic quality across clinical providers, a necessary first step in understanding and improving the diagnostic performance of our health care system.