BIGDATA: F: Collaborative Research: Optimizing Log-Structured-Merge-Based Big Data Management Systems

BIGDATA：F：协作研究：优化基于日志结构合并的大数据管理系统

• 批准号: 1838248
• 负责人: Michael Carey
• 金额: $ 60.48万
• 依托单位: University of California-Irvine
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1838248&HistoricalAwards=false
• 关键词: BIGDATA; Collaborative; Research; Optimizing; Log

Modern big data management systems support fast read and write operations based on the unique identifier (key) of a record. That is, they are fast when inserting key-value pairs, and given a key they quickly return the value associated with that key. To do so, most such systems rely on a Log-Structured-Merge Tree (LSM) structure that batches writes together before writing them to persistent storage. This project will study how to efficiently support more sophisticated operations on LSM-based storage systems, that is, operations that do not simply specify the key of a record. Examples of such operations include searching for records based instead on their location or time. By optimizing the storage and management of big data, this project has the potential to cut the storage costs and energy consumption in data centers. Further, the successful completion of this work will allow users to manage more data with the existing hardware infrastructure, which is critical given the new wave of big data being generated by sensors and the Internet-of-Things. The project will capitalize on the student diversity at two Hispanic Serving Institutions, and thus broaden the participation of under-represented groups in the research process.To support richer data modeling and querying capabilities on top of LSM key-value stores, this project will develop novel LSM indexing and access algorithms to support query plans that utilize both primary and secondary LSM components. In addition, it will design and evaluate flow control policies to dampen or eliminate the notoriously bursty data ingestion behavior that LSM-based storage structures exhibit. It will also study how to automatically and dynamically change LSM compaction policies and parameters based on the query workload. Data-semantics-aware compaction techniques will also be studied. The project will additionally develop novel LSM-aware query optimization techniques; the LSM storage layer is currently treated as a black box by most query optimizers. The planned methods will be deployed and evaluated on the open source Apache AsterixDB system.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代大数据管理系统支持基于记录的唯一标识符（键）的快速读写操作。也就是说，它们在插入键-值对时速度很快，并且给定一个键，它们会快速返回与该键关联的值。要做到这一点，大多数这样的系统依赖于日志结构化合并树（Log-Structured-Merge Tree，LSM）结构，该结构在将写入写入持久存储之前将写入批处理在一起。这个项目将研究如何有效地支持基于LSM的存储系统上更复杂的操作，即不简单地指定记录键的操作。这种操作的示例包括基于记录的位置或时间来搜索记录。通过优化大数据的存储和管理，该项目有可能降低数据中心的存储成本和能源消耗。此外，这项工作的成功完成将使用户能够使用现有的硬件基础设施管理更多的数据，这对于传感器和物联网产生的新一波大数据至关重要。 该项目将利用两个西班牙裔服务机构的学生多样性，从而扩大代表性不足的群体在研究过程中的参与。为了支持更丰富的数据建模和查询功能的LSM键值存储的顶部，该项目将开发新的LSM索引和访问算法，以支持查询计划，利用主要和次要的LSM组件。此外，它还将设计和评估流量控制策略，以抑制或消除基于LSM的存储结构所表现出的臭名昭著的突发数据摄取行为。还将研究如何根据查询工作负载自动动态地更改LSM压缩策略和参数。还将研究数据语义感知压缩技术。该项目还将开发新的LSM感知查询优化技术; LSM存储层目前被大多数查询优化器视为黑盒。计划中的方法将在开源Apache AsterixDB系统上部署和评估。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Design Trade-offs for a Robust Dynamic Hybrid Hash Join

• DOI: 10.14778/3547305.3547327
• 发表时间: 2021-12
• 期刊: Proc. VLDB Endow.
• 作者: Shiva Jahangiri;M. Carey;J. Freytag

On Multi-Valued Indexing in AsterixDB

关于 AsterixDB 中的多值索引

• 发表时间: 2022
• 期刊: co-located with EDBT 2022
• 作者: Galviso, G.;Carey, M.
• 通讯作者: Carey, M.

Multi-valued indexing in Apache AsterixDB (SI DOLAP 2022)

Apache AsterixDB 中的多值索引 (SI DOLAP 2022)

• DOI: 10.1016/j.is.2022.102144
• 发表时间: 2023
• 期刊: Information Systems
• 影响因子: 3.7
• 作者: Galvizo, Glenn;Carey, Michael J.
• 通讯作者: Carey, Michael J.

Breaking Down Memory Walls in LSM-based Storage Systems

打破基于 LSM 的存储系统中的内存墙

• DOI: 10.1145/3318464.3384399
• 发表时间: 2020
• 期刊: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (SIGMOD’20
• 作者: Luo, Chen

Revisiting Runtime Dynamic Optimization for Join Queries in Big Data Management Systems

重新审视大数据管理系统中连接查询的运行时动态优化

• DOI: 10.1145/3604437.3604460
• 发表时间: 2023
• 期刊: ACM SIGMOD Record
• 作者: Pavlopoulou, Christina;Carey, Michael J.;Tsotras, Vassilis J.
• 通讯作者: Tsotras, Vassilis J.