CRII: III: Optimal Data Organization for Hybrid Transactional/Analytical Processing Data Systems

CRII：III：混合事务/分析处理数据系统的最佳数据组织

• 批准号: 1850202
• 负责人: Manos Athanassoulis
• 金额: $ 17.5万
• 依托单位: Trustees of Boston University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-06-01 至 2022-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1850202&HistoricalAwards=false
• 关键词: CRII; III; Optimal; Data; Organization

Scientific, commercial, and governmental applications increasingly rely on data-driven insights and decision-making using both historical data and real-time updates. New workloads are generated by social feeds, sensor readings (a common use-case of Internet-of-Things applications) and electronic micro-payments (an emerging model of e-commerce). They all have in common: (i) a very high volume of transactions and (ii) a high volume of analysis queries that need to use both historic and real-time data to provide useful and actionable insights. The primary challenge is that these workloads have conflicting requirements, and typically use different data systems architectures. On the one hand, we want to be able to answer analysis queries like, "what was the most discussed topic in each month of the past year?", or "what is the average power consumption per neighborhood of city X?". On the other hand, we want to efficiently store incoming updates and be able to provide real-time insights like "where do we have a power network overload now?", or "what is the probability that a disaster is taking place based on the social feeds of a city X?". Traditionally, data systems were engineered to efficiently support either a transactional workload -- that is, storing quickly new items -- or an analytical workload. The latter typically includes changing the data layout and organization, and building auxiliary indexing structures to allow for efficient data access. The emergence of complex workloads has pushed towards the need to develop new systems that can support hybrid transactional/analytical processing (HTAP). This research will allow to execute such workloads efficiently and to anticipate workload changes in a robust way. Ultimately, the project will make data ingestion and data analysis a smoother process and will enable complex applications to have their data analyzed quickly. The researchers will build data systems that can efficiently evaluate mixed workloads by navigating the read-optimized vs. update-optimized continuum of data systems architectures. The key to do so is to vary the physical data organization and find the optimal for each use-case. Typically, data objects are physically organized in various ways between two extremes: either they follow the ingestion order, that is, the way they are generated or inserted in the system, or they are organized based on their value (or a specific subset of their attributes). This "structure" (also called "bounded disorder" in the literature) is treated as a continuum between the two extremes. In-between, hybrid data organizations have different parts of the dataset organized with different schemes. Transactional updates add data with disorder, while answering analytical queries efficiently requires data with bounded disorder. A fundamental challenge today is to find the data organization that enables a data system to offer a tunable balance between efficient updates and fast analysis queries. This project addresses this challenge from three different angles. First, by formulating an optimization problem, which can be solved at run-time. Second, by formulating a robust optimization problem which will deliver good performance even when preliminary assumptions are not accurate. Third, by building access methods that can exploit any inherently limited disorder in the underlying data to reduce the data organization effort needed for efficient analysis tasks. This research effort introduces HTAP data systems that can optimally organize data and exploit inherently bounded disorder while being robust in workload changes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

科学、商业和政府应用程序越来越依赖于使用历史数据和实时更新的数据驱动的见解和决策。新的工作负载是由社交源、传感器读数（物联网应用的常见用例）和电子微支付（电子商务的新兴模式）产生的。他们都有一个共同点：（i）非常高的交易量，以及（ii）需要使用历史和实时数据来提供有用和可操作的见解的大量分析查询。主要挑战是这些工作负载具有相互冲突的需求，并且通常使用不同的数据系统架构。一方面，我们希望能够回答分析查询，比如“过去一年中每个月讨论最多的话题是什么？”或者“X城市每个街区的平均耗电量是多少？".另一方面，我们希望有效地存储传入的更新，并能够提供实时的见解，如“我们现在在哪里有电力网络过载？或者“根据某个城市的社会反馈，灾难发生的概率是多少？".传统上，数据系统被设计为有效地支持事务性工作负载（即快速存储新项目）或分析性工作负载。后者通常包括更改数据布局和组织，以及构建辅助索引结构以允许有效的数据访问。复杂工作负载的出现推动了开发能够支持混合事务/分析处理（HTAP）的新系统的需求。这项研究将允许有效地执行这些工作负载，并以稳健的方式预测工作负载的变化。最终，该项目将使数据摄取和数据分析过程更加顺畅，并使复杂的应用程序能够快速分析数据。研究人员将构建数据系统，通过导航读取优化与更新优化的数据系统架构连续体，可以有效地评估混合工作负载。这样做的关键是改变物理数据组织，并为每个用例找到最佳方案。通常，数据对象以两种极端之间的各种方式进行物理组织：要么遵循摄取顺序，即它们在系统中生成或插入的方式，要么基于它们的值（或其属性的特定子集）进行组织。这种“结构”（在文献中也称为“有界无序”）被视为两个极端之间的连续体。在两者之间，混合数据组织具有使用不同方案组织的数据集的不同部分。transmittance更新添加无序的数据，而有效地回答分析查询需要有界无序的数据。今天的一个基本挑战是找到一种数据组织，使数据系统能够在高效更新和快速分析查询之间提供可调的平衡。本项目从三个不同的角度应对这一挑战。首先，通过制定一个优化问题，它可以在运行时解决。其次，通过制定一个强大的优化问题，即使初步假设不准确，也会提供良好的性能。第三，通过构建访问方法，可以利用底层数据中任何固有的有限无序，以减少高效分析任务所需的数据组织工作。这项研究工作引入了HTAP数据系统，该系统可以最佳地组织数据并利用固有的有界无序，同时在工作量变化中保持稳健。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Building Deletion-Compliant Data Systems

• 发表时间: 2022
• 期刊: IEEE Data Eng. Bull.
• 作者: Manos Athanassoulis;Subhadeep Sarkar;Tarikul Islam Papon;Zichen Zhu;Dimitris Staratzis

Constructing and Analyzing the LSM Compaction Design Space

• DOI: 10.14778/3476249.3476274
• 发表时间: 2021-07
• 期刊: Proc. VLDB Endow.
• 作者: Subhadeep Sarkar;Dimitris Staratzis;Zichen Zhu;Manos Athanassoulis

Endure: A Robust Tuning Paradigm for LSM Trees Under Workload Uncertainty

• DOI: 10.14778/3529337.3529345
• 发表时间: 2021-10
• 期刊: Proc. VLDB Endow.
• 作者: Andrew Huynh;Harshal A. Chaudhari;Evimaria Terzi;Manos Athanassoulis

Compactionary: A Dictionary for LSM Compactions

Compactionary：LSM 压缩字典

• DOI: 10.1145/3514221.3520169
• 发表时间: 2022
• 期刊: SIGMOD '22: Proceedings of the 2022 International Conference on Management of Data
• 作者: Sarkar, Subhadeep;Chen, Kaijie;Zhu, Zichen;Athanassoulis, Manos
• 通讯作者: Athanassoulis, Manos

Query Language Support for Timely Data Deletion

• DOI: 10.48786/edbt.2022.35
• 发表时间: 2022
• 作者: Subhadeep Sarkar;Manos Athanassoulis