Performance Measure and Models for Statistical Prediction

在这个信息时代，我们作为一个社会产生和收集大量的数据。我们面临的挑战是开发合适的和经过验证的分析工具，有效地处理数据，以加快科学发现，加强工程过程和产生有用的信息。为了应对这一挑战，我的重点一直是开发预测性能指标和新的预测算法。** 与C作为测量温度的指标类似，预测性能指标是我们用来评估绝对性能和/或比较不同预测算法的相对性能的指标。 在我的博士研究期间，我定义了一个数量“预测误差”作为删失事件发生时间结果的性能指标。毕业后，我在通过筛查进行癌症检测方面的研究合作使我意识到，人群筛查的目标与医学诊断测试的目标不同，这需要采用与诊断测试不同的性能指标。我研究并提出了一种新的测量方法，称为平均阳性预测值（AP），用于二元分类结局（例如活性化合物或无活性化合物）和删失的二元事件状态数据，该数据来自事件发生时间结局（例如60岁时的无癌症状态）。我在这个研究主题中提出的研究有以下目标：* 比较三个流行的性能指标，包括AP，当新信息被添加到预测算法中时，它们检测预测准确性提高的能力。* 当在抽样框架下招募的研究队列中收集删失生存数据时，开发性能指标的估计量。*扩展AP以评估有序结果的模型预测性能。例如，康复后的中风幸存者可能没有、轻度、中度或重度损伤。准确预测最终结果的能力可以为干预策略和卫生保健计划提供信息。 * 开发新的预测算法是我的另一个研究主题。最近，我的学生成功地模拟了以不规则间隔稀疏收集的纵向数据，就像对人类受试者的研究经常做的那样。我们使用稀疏函数主成分分析（FPCA）的方法来揭示个人的轨迹。在下一阶段，我将开发一种新的算法，通过在基线处结合潜在的生理差异来改善模型对数据的拟合，并更好地预测轨迹。通过理论推导和仿真实验，研究了该算法的性能。预测性能将与FPCA进行比较。** 我的研究计划旨在开发新的算法和性能指标，并加深我们对性能指标的理解。研究计划的产品将对许多领域的科学家、工程师、方法学家和实践者有价值。

In this information era, we as a society generate and collect large amount of data. The challenge is to develop suitable and validated analytic tools that process data effectively and efficiently in order to speed up scientific discovery, enhance engineering process and produce useful information. To tackle this challenge, my focus has been developing prediction performance measures and new prediction algorithms. ******In a similar way to C being a metric measuring temperature, prediction performance measures are metrics that we use to evaluate the absolute performance and/or to compare the relative performance of different prediction algorithms. During my PhD research, I defined a quantity “prediction error” as a performance measure for censored time-to-event outcome. Post-graduation, my research collaboration in cancer detection via screening made me realize that the goal of population screening is different from that of medical diagnostic tests, which necessitates a different performance measure than that used for diagnostic tests. I studied and proposed a new measure named average positive predictive value (AP) for binary categorical outcome (e.g. active compound or not) and the censored binary event status data, which is derived from time-to-event outcome (e.g. cancer-free status at 60 years of age). My proposed research in this research theme have the following goals: *** Compare three popular performance measures including AP on their ability to detect prediction accuracy improvement when new information is being added to a prediction algorithm. *** Develop estimator for performance measures when censored survival data is collected on a study cohort that was recruited under a sampling frame.*** Extend AP to evaluate model prediction performance for ordinal outcomes. For example, a stroke survivor post rehabilitation could have no, mild, moderate or severe impairment. The ability to accurately predict eventual outcome can inform intervention strategy and health care planning. ******Developing new prediction algorithms is my other research theme. Recently my student successfully modeled longitudinal data that have been sparsely collected at irregular intervals, as research on human subjects often do. We have used the sparse function principle component analysis (FPCA) approach to uncover the individual trajectory. In the next phase, I will develop a new algorithm by incorporating the underlying physiological difference at baseline to improve model fit to the data and better predict the trajectory. The properties of the algorithm will be investigated through theoretical derivation and simulation experiment. Prediction performance will be compared to FPCA. ******My research program aims to develop new algorithms and performance measures, and to deepen our understanding of performance measures. The products of research program will be valuable to scientists, engineers, methodologists and practitioners in many fields.