An Incremental Data Management System for Big Data Analytics

大数据分析在科学和工业中具有巨大的重要性。计算资源的广泛可用性；

Big data analytics is of vast importance in science and industry. Widespread availability of compute resources, in the form of cloud computing, gives unprecedented ability to analyze and cross-correlate information from multiple domains, including scientific, social, and corporate data sources. Big data brings new insights and allows optimizations of processes across all aspects of modern enterprise, science, and public services. The abundance of data and its volume creates a pressing need for improved performance during the execution of complex analytical tasks on such data. The aim of this project is to develop technology and a system to support advanced statistical and analytical modelling functionality on top of large (multi-terabyte) data sets. Data analysts typically rely on statistical packages such as R to analyze data and extract accurate models that drive predictions and expose new insights, but such statistical analysis scales poorly to large datasets. Typical analytical workflows on big data are iterative: An analyst builds a model using some data set, then repeatedly refines the model based on the outcome of the previous analysis. These iterative computations become more expensive as data sizes increase, and a single analysis can easily consume many compute-hours of a large cluster. Across such model refinements there exist many opportunity for optimization. For example if the first model and the subsequently refined model, differ by very little, there is no need to recompute the second model from scratch. Adjusting the first model incrementally can offer a precise solution much faster. We wish to make the concept of such incremental computations a first class citizen in our system design. We will develop a system that would be as feature rich as R but at the same time operate on top of ubiquitous data managers supporting incremental computations for improved performance