Efficient query processing and optimizations for big data workloads

计算的每个方面都在经历指数级增长，从传感数据采集吞吐量到处理器能力、存储和带宽。这些指数级的改进正在推动大数据革命。大数据应用程序由以流式方式不断产生的大量数据组成（例如，传感器读数、日志、点击等）。此外，大数据的典型研究和分析工作流程是迭代的。也就是说，使用一些数据参数构建模型，然后使用前一建模阶段的输出迭代地细化。这两个原语，即流数据生成和迭代分析工作流，提供了很多优化的机会。 该项目的目标是探索这些原语，并提供基本算法和技术，以有效地处理和优化大数据工作负载。最终目标是将这些技术包含到端到端数据处理架构中。流数据生成提供了以增量方式维护已经在数据上计算的模型的机会。作为第一示例，可以针对附加在数据集中的新数据递增地维护在数据集上计算的统计算子。此外，已经在数据上计算的模型可以增量地组合（在它们之间或与基础数据）以计算对新建模查询请求的回答。结合两个模型在性能方面可能比从头开始计算一个新模型要上级得多。 在这个项目中，我们计划在我们的系统设计中引入增量计算作为一等公民。我们将逐步维护感兴趣的模型（随着新数据的到来）;通过这些模型的具体化，分析阶段将能够重用先前分析的结果。将制定适当的优化框架，以评估何时以及在何种条件下，这种组合和模型的增量维护是有益的。很明显，后续分析任务的性能将受益于模型的重用和/或广泛的一类模型的组合，探索精确和近似计算。其次，我们计划建立一个包含我们创新的端到端系统。我们的设计将集中在用于统计处理和数据分析的流行语言上，以表达建模工作负载（例如，R）和合适的系统基础设施来实现和执行我们的框架。 我们研究的最终产品将是一个包含所有研究的系统，利用熟悉的分析查询处理接口（如R）提供非常快速的大数据分析。这样的系统将有助于数据科学家在所需时间的一小部分内进行高级研究，因为它能够以增量的方式无缝地重用和共享结果。

Every aspect of computing has been experiencing exponential growth, from sensory data acquisition throughput to processor power, storage and bandwidth. These exponential improvements are enabling the big data revolution. Big data applications consist of volumes of data that are constantly produced in a streaming fashion (e.g., sensor readings, logs, click-through etc.). In addition typical research and analysis workflows on big data are iterative. Namely a model is built using some data parameters, then iteratively refined using the output of the previous modeling phase. Both such primitives, namely streaming data generation and iterative analysis workflows, provide a lot of opportunity for optimizations. The goal of this project is to explore these primitives and deliver fundamental algorithms and techniques to efficiently process and optimize big data workloads. The end goal is to encompass such techniques into end-to-end data processing architecture. Streaming data generation provides the opportunity to maintain models already computed on the data in an incremental fashion. As a first example, a statistical operator computed on a data set can be incrementally maintained for new data appended in the data set. In addition, models already computed on the data can be combined (among themselves or with base data) incrementally to compute answers to new modeling query requests. Combining two models could be vastly superior in terms of performance than computing a new model from scratch. In this project we plan to introduce incremental computations as a first class citizen in our system design. We will incrementally maintain models of interest (as new data arrive); via materialization of such models, analysis phases will be able to re-use results available from prior analysis. Suitable optimization frameworks will be developed to assess when and under what conditions such combinations and incremental maintenance of models is beneficial. It is evident that the performance of subsequent analysis tasks will benefit from model re-use and/or combination for a wide class of models exploring both exact and approximate computations. Second, we plan to build an end-to-end system encompassing our innovations. Our design will be centered on popular languages for statistical processing and data analysis to express modeling workloads (e.g., R) and the suitable systems infrastructure to implement and execute our framework. The end product of our research will be a system encompassing all of the research conducted delivering very fast big data analytics utilizing familiar analytical query processing interfaces such as R. Such a system will benefit and help data scientists conduct advanced research in a fraction of the time required, by being able to seamlessly re-use and share results in an incremental fashion.