EVALUATION OF PERFORMANCE MEASURES FOR PREDICTIVE MODELS

二元医疗结果的预测数学模型

Predictive mathematical models of dichotomous medical outcomes that generate an estimated probability that a patient will have an outcome are central to many clinical decision aids and are fundamental to comparative analyses of medical care based on risk- adjusted events. In such applications, inaccurate assessment of patient risk can have significant health care and health policy implications. Thus it is critical that researchers have methods for assessing the accuracy of the predictions from these models. Statistical methods for assessing the overall performance, calibration, and discrimination of these predictive models have been proposed and are currently being used. Unfortunately, the lack of a meaningful interpretation and thorough understanding by researchers of these measures of predictive performance limit their usefulness and the ability of researchers to evaluate and compare predictive models. Using data from several large controlled randomized trials, and previously developed simulation programs, this project will provide an evaluation of the appropriate use of these measures by addressing three questions: 1. What is the expected agreement between performance measures? 2. What is the sensitivity of each performance measure to model and data characteristics? 3. What is a meaningful interpretation of the calculated value for each statistic? these questions will be addressed in four phases. In phase one the mathematical formula used for each measure will be reformulated and compared. In phase two these performance measures will be applied and compared for existing predictive models. In phase three the performance measures will be applied to simulated data and predictive models. Based on the results of the prior phases, and input from clinical researchers, in phase four useful interpretations of the meaning and utility of each measure will be evaluated. This study should broaden the understanding of these measures of predictive performance for models of dichotomous outcomes, and improve the ability of health policy and health services researchers to properly assess the accuracy, and thus utility, of predictive models.