III: Small: Rethinking the Data Organization and Lifecycle in LSM Storage Systems

III：小：重新思考 LSM 存储系统中的数据组织和生命周期

基本信息

批准号：
2227669
负责人：
Evangelos Christidis
金额：
$ 60万
依托单位：
University of California-Riverside
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2023
资助国家：
美国
起止时间：
2023-01-01 至 2025-12-31
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2227669&HistoricalAwards=false
关键词：
III Small Rethinking Data Organization

项目摘要

To support the efficient storing of large amounts of data, many modern database systems use the Log Structured Merge tree (LSM) technology. This technology allows grouping many data updates together, before applying them to the database. This project has identified several limitations of LSM storage, which cause reduced rates of reads and writes to the database system. Specifically, current LSM systems do not consider the hotness of a data record when deciding how to store it, and may also suffer from periodic stalls, where the system may become unresponsive while large maintenance operations, called merges, are performed. Further, LSM systems are inefficient at exploiting larger computer memories. The developed techniques create novel data organization and flow patterns in the LSM storage, which leverage modern hardware capabilities to boost the read and write capabilities of the storage system. Improving the performance of database systems will allow storing larger data at lower costs, thus making storage systems more accessible to scientists and general users. This project will also strengthen and extend the ongoing undergraduate research and high school outreach activities of the investigators. The project has several research aims. First, algorithms will be developed to store frequently accessed records in more accessible locations for faster retrieval. This will facilitate a bi-directional LSM tree architecture, where records flow both top-down and bottom-up. This will allow naturally maintaining hot records together, for faster querying and more effective caching. Second, new algorithms will be created to improve the speed of data merges. Periodic merges are used to maintain the stored data organized and consistent. This aim will study how to universally partition LSM runs to facilitate splitting a large merge into multiple disjoint sub-merges, thus reducing stall periods. The third aim will create algorithms to better utilize large memory sizes and multithreading parallelism, and develop a mixed memory-disk LSM tree. The key idea is that, instead of directly enlarging the MemTable, where recent writes are buffered, some components can be pinned in memory, with a more efficient organization, and enable parallel execution on queries. The project includes theoretical analysis, experimental study and software development. The developed algorithms are tested on real-world data and integrated in real database systems. This integration may increase the impact of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

为了支持大量数据的有效存储，许多现代数据库系统都使用日志结构合并树（LSM）技术。在将许多数据更新应用于数据库之前，该技术允许将许多数据更新分组在一起。该项目已经确定了LSM存储的几个局限性，这导致读取率降低，并写入数据库系统。具体而言，当前的LSM系统在决定如何存储数据时不会考虑数据记录的热度，并且可能会遭受定期摊位的折磨，在该摊位中，该系统可能会在执行大型维护操作（称为合并）的情况下变得无响应。此外，LSM系统在利用较大的计算机记忆方面效率低下。开发的技术在LSM存储中创建了新颖的数据组织和流程模式，该技术利用现代硬件功能来增强存储系统的读写功能。提高数据库系统的性能将使较大的数据以较低的成本存储，从而使存储系统更容易被科学家和普通用户访问。该项目还将加强和扩展正在进行的研究人员的研究生研究和高中外展活动。该项目有一些研究目标。首先，将开发算法以存储在更可访问的位置存储经常访问的记录，以更快地检索。这将有助于双向LSM树建筑，其中记录在自上而下和自下而上。这将允许自然地保持热记录，以进行更快的查询和更有效的缓存。其次，将创建新的算法以提高数据合并速度。定期合并用于维护储存的数据有组织和一致。该目标将研究如何普遍分区LSM运行以促进将大型合并分成多个不相交的子网状物，从而减少失速时期。第三个目标将创建算法以更好地利用大型内存大小和多线程并行性，并开发混合的内存磁盘LSM树。关键的想法是，与其直接放大记忆的作品，而最近的写作是缓冲的，而是可以通过更有效的组织将某些组件固定在内存中，并可以对查询进行并行执行。该项目包括理论分析，实验研究和软件开发。在现实世界数据上测试了开发的算法并将其集成到实际数据库系统中。这种整合可能会增加项目的影响。该奖项反映了NSF的法定任务，并通过使用基金会的知识分子优点和更广泛的影响审查标准来评估值得支持。