Performance Measure and Models for Statistical Prediction

在这个信息时代，我们作为一个社会产生和收集了大量的数据。挑战是开发适当的和经过验证的分析工具，有效和高效地处理数据，以加快科学发现、增强工程过程和产生有用的信息。为了应对这一挑战，我的重点一直是开发预测性能指标和新的预测算法。 与C作为测量温度的度量类似，预测性能度量是我们用来评估绝对性能和/或比较不同预测算法的相对性能的度量。在我的博士研究期间，我定义了一个量化的“预测误差”，作为对经过审查的事件间隔结果的性能度量。毕业后，我在筛查癌症检测方面的研究合作让我意识到，人群筛查的目标与医学诊断测试的目标不同，这就需要一种与诊断测试不同的绩效衡量标准。我研究并提出了一种新的测量方法，名为平均阳性预测值(AP)，用于二元分类结果(例如，是否有活性化合物)和经审查的二元事件状态数据，该数据来自事件发生时间结果(例如，60岁时的无癌症状态)。我在这一研究主题中提出的研究有以下目标： 比较三种流行的性能指标，包括AP，当新信息被添加到预测算法时，它们检测预测精度改进的能力。 在抽样框架下招募的研究队列上收集经审查的生存数据时，开发绩效衡量的估计器。 扩展AP以评估模型对顺序结果的预测性能。例如，康复后的中风幸存者可能没有、轻度、中度或严重的损害。准确预测最终结果的能力可以为干预策略和卫生保健计划提供信息。 开发新的预测算法是我的另一个研究主题。最近，我的学生成功地对以不规则间隔稀疏收集的纵向数据进行了建模，就像对人类受试者的研究经常做的那样。我们使用了稀疏函数主成分分析(FPCA)方法来揭示个体的轨迹。在下一阶段，我将开发一种新的算法，通过结合基线上的潜在生理差异来改进模型对数据的拟合，并更好地预测轨迹。通过理论推导和仿真实验研究了该算法的性能。预测性能将与FPCA进行比较。 我的研究计划旨在开发新的算法和性能衡量标准，并加深我们对绩效衡量标准的理解。研究计划的成果将对许多领域的科学家、工程师、方法学家和实践者具有很高的价值。

In this information era, we as a society generate and collect large amount of data. The challenge is to develop suitable and validated analytic tools that process data effectively and efficiently in order to speed up scientific discovery, enhance engineering process and produce useful information. To tackle this challenge, my focus has been developing prediction performance measures and new prediction algorithms. In a similar way to C being a metric measuring temperature, prediction performance measures are metrics that we use to evaluate the absolute performance and/or to compare the relative performance of different prediction algorithms. During my PhD research, I defined a quantity “prediction error” as a performance measure for censored time-to-event outcome. Post-graduation, my research collaboration in cancer detection via screening made me realize that the goal of population screening is different from that of medical diagnostic tests, which necessitates a different performance measure than that used for diagnostic tests. I studied and proposed a new measure named average positive predictive value (AP) for binary categorical outcome (e.g. active compound or not) and the censored binary event status data, which is derived from time-to-event outcome (e.g. cancer-free status at 60 years of age). My proposed research in this research theme have the following goals: Compare three popular performance measures including AP on their ability to detect prediction accuracy improvement when new information is being added to a prediction algorithm. Develop estimator for performance measures when censored survival data is collected on a study cohort that was recruited under a sampling frame. Extend AP to evaluate model prediction performance for ordinal outcomes. For example, a stroke survivor post rehabilitation could have no, mild, moderate or severe impairment. The ability to accurately predict eventual outcome can inform intervention strategy and health care planning. Developing new prediction algorithms is my other research theme. Recently my student successfully modeled longitudinal data that have been sparsely collected at irregular intervals, as research on human subjects often do. We have used the sparse function principle component analysis (FPCA) approach to uncover the individual trajectory. In the next phase, I will develop a new algorithm by incorporating the underlying physiological difference at baseline to improve model fit to the data and better predict the trajectory. The properties of the algorithm will be investigated through theoretical derivation and simulation experiment. Prediction performance will be compared to FPCA. My research program aims to develop new algorithms and performance measures, and to deepen our understanding of performance measures. The products of research program will be valuable to scientists, engineers, methodologists and practitioners in many fields.