SBIR Phase II: A Multithreaded Storage Engine Using Highly-Concurrent Fractal Trees

SBIR第二阶段：使用高并发分形树的多线程存储引擎

• 批准号: 1058565
• 负责人: Bradley Kuszmaul
• 金额: $ 50万
• 依托单位: Tokutek, Inc.
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2011
• 资助国家: 美国
• 起止时间: 2011-02-01 至 2013-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1058565&HistoricalAwards=false
• 关键词: SBIR; Phase; II; Multithreaded; Storage

This Small Business Innovation Research (SBIR) Phase II project will apply multithreading techniques to provide multi-terabyte (and larger) high-performance databases in MySQL. The company has developed a highperformance storage engine for MySQL, which maintains indexes on live data 100 times faster than current commonly-used structures. The technology solves the problem of maintaining indexes on large databases in the face of high trickle-load indexing rates. In Phase I, the company developed a multithreaded bulk loader to solve the problem of how to load data quickly. The next significant research problems for large MySQL databases are to allow online, or "hot", schema changes in which, for example, an index can be added without taking the database down, and to use multithreading to speed up joins and reductions so that the large data sets can be queried quickly. In this project, the researchers will investigate the use of multithreading to support hot indexing and parallel joins reductions.If successful, multi-terabyte and larger databases will be manageable and fast on modest hardware, and the hardware will be scalable both with CPU cores and disks. The broader impact of this work is driven by faster, cheaper, lower-power on-disk storage. Organizations that have very large databases will be able to use much less hardware, both saving money and reducing power consumption significantly. Currently many application areas do not employ databases because their performance is too slow. Speeding up databases by two orders-of-magnitude can help grow the market. Currently, many organizations fail to make good use of the data they have collected because they cannot manage it, index it, or query it fast enough to be useful. Applications in finance, retail, homeland security, telecommunications, and scientific computing will benefit from improved manageability and performance. As users' appetite for data continues to outstrip the availability of fast memory, organizing multithreaded queries on disk-based data for performance will continue to grow in importance.

这个小型企业创新研究 (SBIR) 第二阶段项目将应用多线程技术在 MySQL 中提供多 TB（和更大）的高性能数据库。该公司为 MySQL 开发了高性能存储引擎，其维护实时数据索引的速度比当前常用结构快 100 倍。该技术解决了在高涓流负载索引率的情况下维护大型数据库索引的问题。第一阶段，公司开发了多线程批量加载器，解决如何快速加载数据的问题。大型 MySQL 数据库的下一个重要研究问题是允许在线或“热”模式更改，例如，可以在不关闭数据库的情况下添加索引，并使用多线程来加速联接和归约，以便可以快速查询大型数据集。在这个项目中，研究人员将研究使用多线程来支持热索引和并行连接减少。如果成功，数 TB 和更大的数据库将可以在适度的硬件上快速管理，并且硬件将可以通过 CPU 核心和磁盘进行扩展。这项工作的更广泛影响是由更快、更便宜、更低功耗的磁盘存储驱动的。拥有非常大的数据库的组织将能够使用更少的硬件，既节省资金又显着降低功耗。目前很多应用领域没有使用数据库，因为数据库的性能太慢。将数据库速度提高两个数量级可以帮助扩大市场。目前，许多组织未能充分利用他们收集的数据，因为他们无法足够快地管理数据、索引数据或查询数据以使其发挥作用。金融、零售、国土安全、电信和科学计算领域的应用程序将受益于改进的可管理性和性能。 随着用户对数据的需求继续超出快速内存的可用性，组织基于磁盘的数据的多线程查询以提高性能将变得越来越重要。