SBIR Phase II: A Multithreaded Storage Engine Using Highly-Concurrent Fractal Trees

这个小型企业创新研究(SBIR)第二阶段项目将应用多线程技术，在MySQL中提供数TB(以及更大)的高性能数据库。该公司已经为MySQL开发了一个高性能存储引擎，它维护实时数据的索引的速度比目前常用的结构快100倍。该技术解决了在高涓流负载索引率的情况下维护大型数据库上的索引的问题。在第一阶段，该公司开发了多线程批量加载器，以解决如何快速加载数据的问题。大型MySQL数据库的下一个重要研究问题是允许在线或“热”模式更改，例如，可以在不关闭数据库的情况下添加索引，以及使用多线程来加快联接和缩减速度，以便快速查询大型数据集。在这个项目中，研究人员将研究使用多线程来支持热索引和并行连接缩减。如果成功，数TB和更大的数据库将可以在适度的硬件上进行快速管理，并且硬件将通过CPU核心和磁盘进行扩展。这项工作的更广泛影响是由更快、更便宜、更低功耗的磁盘存储推动的。拥有非常大数据库的组织将能够使用更少的硬件，既节省了资金，又显著降低了功耗。目前，许多应用领域不使用数据库，因为它们的性能太慢。将数据库速度提高两个数量级可以帮助市场增长。目前，许多组织无法很好地利用他们收集的数据，因为他们无法管理、索引或查询足够快的有用数据。金融、零售、国土安全、电信和科学计算领域的应用将受益于改进的可管理性和性能。随着用户对数据的需求继续超过快速内存的可用性，组织对基于磁盘的数据的多线程查询以获得性能将继续变得越来越重要。

This Small Business Innovation Research (SBIR) Phase II project will apply multithreading techniques to provide multi-terabyte (and larger) high-performance databases in MySQL. The company has developed a highperformance storage engine for MySQL, which maintains indexes on live data 100 times faster than current commonly-used structures. The technology solves the problem of maintaining indexes on large databases in the face of high trickle-load indexing rates. In Phase I, the company developed a multithreaded bulk loader to solve the problem of how to load data quickly. The next significant research problems for large MySQL databases are to allow online, or "hot", schema changes in which, for example, an index can be added without taking the database down, and to use multithreading to speed up joins and reductions so that the large data sets can be queried quickly. In this project, the researchers will investigate the use of multithreading to support hot indexing and parallel joins reductions.If successful, multi-terabyte and larger databases will be manageable and fast on modest hardware, and the hardware will be scalable both with CPU cores and disks. The broader impact of this work is driven by faster, cheaper, lower-power on-disk storage. Organizations that have very large databases will be able to use much less hardware, both saving money and reducing power consumption significantly. Currently many application areas do not employ databases because their performance is too slow. Speeding up databases by two orders-of-magnitude can help grow the market. Currently, many organizations fail to make good use of the data they have collected because they cannot manage it, index it, or query it fast enough to be useful. Applications in finance, retail, homeland security, telecommunications, and scientific computing will benefit from improved manageability and performance. As users' appetite for data continues to outstrip the availability of fast memory, organizing multithreaded queries on disk-based data for performance will continue to grow in importance.