Efficient query processing and optimizations for big data workloads

点击翻译按钮获取中文摘要

Every aspect of computing has been experiencing exponential growth, from sensory data acquisition throughput to processor power, storage and bandwidth. These exponential improvements are enabling the big data revolution. Big data applications consist of volumes of data that are constantly produced in a streaming fashion (e.g., sensor readings, logs, click-through etc.). In addition typical research and analysis workflows on big data are iterative. Namely a model is built using some data parameters, then iteratively refined using the output of the previous modeling phase. Both such primitives, namely streaming data generation and iterative analysis workflows, provide a lot of opportunity for optimizations. ******The goal of this project is to explore these primitives and deliver fundamental algorithms and techniques to efficiently process and optimize big data workloads. The end goal is to encompass such techniques into end-to-end data processing architecture. Streaming data generation provides the opportunity to maintain models already computed on the data in an incremental fashion. As a first example, a statistical operator computed on a data set can be incrementally maintained for new data appended in the data set. In addition, models already computed on the data can be combined (among themselves or with base data) incrementally to compute answers to new modeling query requests. Combining two models could be vastly superior in terms of performance than computing a new model from scratch. ******In this project we plan to introduce incremental computations as a first class citizen in our system design. We will incrementally maintain models of interest (as new data arrive); via materialization of such models, analysis phases will be able to re-use results available from prior analysis. Suitable optimization frameworks will be developed to assess when and under what conditions such combinations and incremental maintenance of models is beneficial. It is evident that the performance of subsequent analysis tasks will benefit from model re-use and/or combination for a wide class of models exploring both exact and approximate computations. Second, we plan to build an end-to-end system encompassing our innovations. Our design will be centered on popular languages for statistical processing and data analysis to express modeling workloads (e.g., R) and the suitable systems infrastructure to implement and execute our framework. ******The end product of our research will be a system encompassing all of the research conducted delivering very fast big data analytics utilizing familiar analytical query processing interfaces such as R. Such a system will benefit and help data scientists conduct advanced research in a fraction of the time required, by being able to seamlessly re-use and share results in an incremental fashion.**