III: Small: Task-aware Materialization for Fast Data Analytics

数据到知识的管道是我们这个数据驱动的世界的核心。它使用原始格式的数据，然后清理、转换、集成、存储、处理和分析数据，以获取可用格式的知识。数据分析管道很复杂，它们的工作流通常由几个更简单的任务组成，这些任务链接在一起。为了加快这类流水线的速度，一种普遍存在的优化技术是将任务的中间结果具体化，以便下游任务能够尽可能高效地访问中间数据。现有的物化技术存在几个缺点，包括在存储和预处理时间方面的成本过高。为了解决这些缺陷，该提案将开发能够显著提高数据分析应用程序性能的智能物化算法。因此，它将使数据科学家能够更快地获得可操作的见解，并将影响生物学、经济学、社会学和医学等领域的研究。该项目的目标是使用三个新想法设计、实施和评估物化技术：任务感知的物化；关于物化什么和如何物化的细粒度决策；以及可以用空间换取时间的多个设计点，以实现数据分析管道的最佳性能。该项目还将探讨如何设计物化数据结构，以适应数据和下游工作负荷的变化。从理论上讲，本研究旨在为不同任务的物化在空间和时间之间的权衡提供理论保障。从实践的角度来看，它将导致实施和评估在现实世界数据分析应用程序上开发的算法，包括通过谱系跟踪、统计推理、图形模式检索和社交网络分析实现可视化。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The data-to-knowledge pipeline is central to our data-driven world. It consumes data in raw format and then cleans, transforms, integrates, stores, processes, and analyzes the data to obtain knowledge in a usable format. Data analytics pipelines are complex and their workflows typically consist of several simpler tasks chained together. To speed up such pipelines, a ubiquitous optimization technique is to materialize the intermediate result of a task, so that downstream tasks can access the intermediate data as efficiently as possible. Existing materialization techniques suffer from several drawbacks, including prohibitively large cost in terms of storage and preprocessing time. To address these drawbacks, this proposal will develop smart materialization algorithms that can significantly accelerate the performance of data analytics applications. As a result, it will enable data scientists to obtain actionable insights faster and will impact research in areas such as biology, economics, sociology and the medical sciences. The goal of this project is to design, implement and evaluate materialization techniques using three novel ideas: task-aware materialization; fine-grained decisions on what and how to materialize; and multiple design points that can trade off space for time to achieve optimal performance of data analytics pipelines. The project will also explore how to design data structures for materialization that are adaptive to changes in the data and the downstream workload. From a theoretical viewpoint, the proposed research will aim to obtain theoretical guarantees on the tradeoff between space and time for materialization for different tasks. From a practical viewpoint, it will result in implementing and evaluating the developed algorithms on real-world data analytics applications, including visualization through lineage tracking, statistical inference, pattern retrieval in graphs and social network analysis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.