Incremental Regression Analysis of Streaming Data: Estimating Function Theory and Applications

分布式数据存储和并行计算系统(如ApacheSpark)的出现为数据分析和建模提供了创新机会。该项目专注于星火Lambda架构下的流数据回归分析，旨在开发一个新的大数据分析工具箱。流数据是指按顺序到达的一系列数据批次。由于许多人工智能增强型医疗设备的蓬勃发展，此类数据收集方案最近在生物医学领域变得丰富起来，这些设备旨在监控智能个性化产品提供的医疗的安全性和有效性，或测量实时生理变量，如心跳、体温和身体活动。这种所谓的深度表型技术，无论是从体量还是从速度上都极大地改变了信息获取的方式。作为最重要的数据分析，回归分析将在拟议的项目中重建，以应对流动数据处理带来的各种挑战。由此产生的方法可以应用于许多实际领域，其中对数据流的增量学习是主要感兴趣的。该项目的总体目标是开发一种增量统计推断，以解决使用存储在Spark的Lambda架构中的流数据进行回归分析的方法挑战。高效的增量方法不需要使用任何历史原始数据，而只需要使用历史汇总统计数据和新到达的数据批次。在这个项目完成后，PI期望做出以下新的贡献：(I)在估计函数的背景下开发可更新估计和增量推理的新理论；(Ii)开发扩展的速度数据流结构，称为RHO结构，其中增加了一个新的层来进行与推理相关的量的更新，例如Fisher信息量；(Iii)将所提出的方法应用于许多重要的回归模型，例如广义线性模型、广义估计方程(GEE)、Cox比例风险模型和分位数回归模型。巨蟒和R包都将从这个项目中交付给公众。这个奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The advent of distributed data storage and parallel computing systems such as the Apache Spark has provided opportunities of innovation in data analytics and modeling. This project focuses on regression analysis of streaming data under the Spark's Lambda architecture, aiming to develop a new toolbox of Big Data analytics. Streaming data refers to a series of data batches that arrives sequentially. Such data collection schemes have become abundant lately in biomedical fields due to the booming of many AI-enhanced medical devices that are designed to monitor safety and effectiveness of medical treatments delivered by smart personalized products, or to measure real-time physiological variables such as heart beats, body temperature, and physical activity. This so-called deep phenotyping technology has significantly changed the way of information acquisition in terms of both volume and velocity. Being the most important data analytics, the regression analysis will be rebuilt in the proposed project to address various challenges from the processing of streaming data. The resulting methodology may be applied to many practical fields, where incremental learning with data streams is of primary interest. The overarching goal of this project is to develop an incremental statistical inference to address methodological challenges in regression analysis with streaming data stored in the Spark's Lambda architecture. Efficient incremental methodology requires no use of any historic raw data, rather only historic summary statistics and a newly arrived data batch. At the completion of this project the PI expects to make the following new contributions: (i) To develop a new theory of renewable estimation and incremental inference in the context of estimating functions; (ii) to develop an expansion of speed data flow architecture, called the Rho architecture, in which a new layer is added to carry over updates of inference-related quantities such as the Fisher information; (iii) to apply the proposed methodology in many important regression models, such as the generalized linear models, the generalized estimating equations (GEE), the Cox proportional hazards model, and the quantile regression model. Both python and R packages will be delivered from this project to the public.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.