SBIR Phase II: A Multithreaded Storage Engine Using Highly-Concurrent Fractal Trees

这个小企业创新研究（SBIR）第二阶段项目将应用多线程技术在MySQL中提供多tb（甚至更大）的高性能数据库。该公司为MySQL开发了一种高性能存储引擎，它在实时数据上维护索引的速度比目前常用的结构快100倍。该技术解决了面对高涓流负载索引率时在大型数据库上维护索引的问题。在第一阶段，该公司开发了一个多线程批量加载器来解决如何快速加载数据的问题。对于大型MySQL数据库来说，下一个重要的研究问题是允许在线或“热”模式更改，例如，可以在不关闭数据库的情况下添加索引，并使用多线程来加速连接和约简，以便可以快速查询大型数据集。在这个项目中，研究人员将研究使用多线程来支持热索引和并行连接减少。如果成功，多tb和更大的数据库将在中等硬件上易于管理和快速，并且硬件将可以通过CPU内核和磁盘进行扩展。这项工作的广泛影响是由更快、更便宜、更低功耗的磁盘存储驱动的。拥有大型数据库的组织将能够使用更少的硬件，既节省了资金，又显著降低了功耗。目前，由于数据库的性能太慢，许多应用领域没有使用数据库。将数据库的速度提高两个数量级有助于扩大市场。目前，许多组织不能很好地利用他们收集的数据，因为他们不能对其进行管理、编制索引或快速查询。金融、零售、国土安全、电信和科学计算领域的应用将受益于改进的可管理性和性能。随着用户对数据的需求继续超过快速内存的可用性，在基于磁盘的数据上组织多线程查询以提高性能将变得越来越重要。

This Small Business Innovation Research (SBIR) Phase II project will apply multithreading techniques to provide multi-terabyte (and larger) high-performance databases in MySQL. The company has developed a highperformance storage engine for MySQL, which maintains indexes on live data 100 times faster than current commonly-used structures. The technology solves the problem of maintaining indexes on large databases in the face of high trickle-load indexing rates. In Phase I, the company developed a multithreaded bulk loader to solve the problem of how to load data quickly. The next significant research problems for large MySQL databases are to allow online, or "hot", schema changes in which, for example, an index can be added without taking the database down, and to use multithreading to speed up joins and reductions so that the large data sets can be queried quickly. In this project, the researchers will investigate the use of multithreading to support hot indexing and parallel joins reductions.If successful, multi-terabyte and larger databases will be manageable and fast on modest hardware, and the hardware will be scalable both with CPU cores and disks. The broader impact of this work is driven by faster, cheaper, lower-power on-disk storage. Organizations that have very large databases will be able to use much less hardware, both saving money and reducing power consumption significantly. Currently many application areas do not employ databases because their performance is too slow. Speeding up databases by two orders-of-magnitude can help grow the market. Currently, many organizations fail to make good use of the data they have collected because they cannot manage it, index it, or query it fast enough to be useful. Applications in finance, retail, homeland security, telecommunications, and scientific computing will benefit from improved manageability and performance. As users' appetite for data continues to outstrip the availability of fast memory, organizing multithreaded queries on disk-based data for performance will continue to grow in importance.