An Incremental Data Management System for Big Data Analytics

大数据分析在科学和工业中具有巨大的重要性。计算资源的广泛可用性， 以云计算的形式，提供了前所未有的分析和交叉关联信息的能力， 多个领域，包括科学、社会和企业数据源。大数据带来了新的见解， 允许在现代企业、科学和公共服务的各个方面优化流程。 大量的数据和数据量迫切需要在 对这些数据执行复杂的分析任务。该项目的目的是开发技术和 系统，以支持在大型（多兆字节） 数据集。数据分析师通常依赖R等统计软件包来分析数据并提取准确的 驱动预测并揭示新见解的模型，但这种统计分析的规模很小， 数据集。大数据的典型分析工作流程是迭代的：分析师使用一些数据集构建模型， 然后根据先前分析的结果反复改进模型。这些迭代计算 随着数据大小的增加，成本会变得更高，而且一次分析很容易消耗许多计算时间 一个大的集群。在这种模型改进中，存在许多优化的机会。例如如果 第一个模型和随后改进的模型差别很小，不需要重新计算 第二个模型从零开始逐步调整第一个模型可以更快地提供精确的解决方案。 我们希望在我们的系统设计中将这种增量计算的概念作为一等公民。 我们将开发一个像R一样功能丰富的系统，但同时在无处不在的基础上运行。 数据管理器支持增量计算，以提高性能

Big data analytics is of vast importance in science and industry. Widespread availability of compute resources, in the form of cloud computing, gives unprecedented ability to analyze and cross-correlate information from multiple domains, including scientific, social, and corporate data sources. Big data brings new insights and allows optimizations of processes across all aspects of modern enterprise, science, and public services. The abundance of data and its volume creates a pressing need for improved performance during the execution of complex analytical tasks on such data. The aim of this project is to develop technology and a system to support advanced statistical and analytical modelling functionality on top of large (multi-terabyte) data sets. Data analysts typically rely on statistical packages such as R to analyze data and extract accurate models that drive predictions and expose new insights, but such statistical analysis scales poorly to large datasets. Typical analytical workflows on big data are iterative: An analyst builds a model using some data set, then repeatedly refines the model based on the outcome of the previous analysis. These iterative computations become more expensive as data sizes increase, and a single analysis can easily consume many compute-hours of a large cluster. Across such model refinements there exist many opportunity for optimization. For example if the first model and the subsequently refined model, differ by very little, there is no need to recompute the second model from scratch. Adjusting the first model incrementally can offer a precise solution much faster. We wish to make the concept of such incremental computations a first class citizen in our system design. We will develop a system that would be as feature rich as R but at the same time operate on top of ubiquitous data managers supporting incremental computations for improved performance