Performance Measure and Models for Statistical Prediction

点击翻译按钮获取中文摘要

In this information era, we as a society generate and collect large amount of data. The challenge is to develop suitable and validated analytic tools that process data effectively and efficiently in order to speed up scientific discovery, enhance engineering process and produce useful information. To tackle this challenge, my focus has been developing prediction performance measures and new prediction algorithms. ******In a similar way to C being a metric measuring temperature, prediction performance measures are metrics that we use to evaluate the absolute performance and/or to compare the relative performance of different prediction algorithms. During my PhD research, I defined a quantity “prediction error” as a performance measure for censored time-to-event outcome. Post-graduation, my research collaboration in cancer detection via screening made me realize that the goal of population screening is different from that of medical diagnostic tests, which necessitates a different performance measure than that used for diagnostic tests. I studied and proposed a new measure named average positive predictive value (AP) for binary categorical outcome (e.g. active compound or not) and the censored binary event status data, which is derived from time-to-event outcome (e.g. cancer-free status at 60 years of age). My proposed research in this research theme have the following goals: *** Compare three popular performance measures including AP on their ability to detect prediction accuracy improvement when new information is being added to a prediction algorithm. *** Develop estimator for performance measures when censored survival data is collected on a study cohort that was recruited under a sampling frame.*** Extend AP to evaluate model prediction performance for ordinal outcomes. For example, a stroke survivor post rehabilitation could have no, mild, moderate or severe impairment. The ability to accurately predict eventual outcome can inform intervention strategy and health care planning. ******Developing new prediction algorithms is my other research theme. Recently my student successfully modeled longitudinal data that have been sparsely collected at irregular intervals, as research on human subjects often do. We have used the sparse function principle component analysis (FPCA) approach to uncover the individual trajectory. In the next phase, I will develop a new algorithm by incorporating the underlying physiological difference at baseline to improve model fit to the data and better predict the trajectory. The properties of the algorithm will be investigated through theoretical derivation and simulation experiment. Prediction performance will be compared to FPCA. ******My research program aims to develop new algorithms and performance measures, and to deepen our understanding of performance measures. The products of research program will be valuable to scientists, engineers, methodologists and practitioners in many fields.