III-COR - ChunkyStore: Physical Database Design for Next-Generation Databases

ChunkyStore：下一代数据库的物理数据库设计传统数据库以行为主格式在磁盘上布局表，但最近的研究表明，其他物理布局（例如，使用列为主格式）可以在许多应用程序中产生实质性（数量级）的性能提升。ChunkyStore项目的主要目标是探索物理存储布局和数据库性能之间的关系。这样做的主要机制是将数据库表划分为由不同数量的共存行和列组成的“块”，并随着时间的推移自适应地调整这些块以优化性能。实现这一目标的具体技术包括：(1)研究不同“分块”算法在不同工作负载和应用程序下的性能，包括科学数据管理和数据仓库；(2)构建基于历史工作负载模式模型选择最佳分块表的自动存储设计器；(3)研究基于块的非表格数据的表示和布局，如图像数组（这在科学应用中尤为重要）；(4)利用存储设计器、阵列管理技术以及各种复制和分发策略，在大型数据集上部署高性能的软件系统。ChunkyStore将大大提高数据库系统的性能，并证明它们可以用于管理各种科学数据。教育和传播工作将通过研究论文、开放源代码发布、几个用户研讨会以及与麻省理工学院研究生数据库系统课程项目的整合来进行。有关ChunkyStore的更多信息，包括出版物和源代码，请参阅http://db.csail.mit.edu/chunkystore/。

ChunkyStore: Physical Database Design for Next-Generation DatabasesTraditional databases lay out tables on disk in a row-major format, but recent research has shown that other physical layouts (e.g., using a column-major format) can yield substantial (order-of-magnitude) performance gains in many applications. The broad goal of the ChunkyStore project is to explore this relationship between physical storage layouts and database performance. The primary mechanism for doing this is to partition database tables into "chunks" of varying numbers of co-located rows and columns and adaptively adjusting these chunks over time to optimize performance.Specific techniques used to achieve this goal include: (1) investigation of the performance of different "chunking" algorithms under different workloads and applications, including scientific data management and data warehousing; (2) building an automatic storage designer that chooses the best chunking of tables based on models of historical workload patterns; (3) studying chunk-based representations and layout of non-tabular data, such as arrays of imagery (which are particularly important in scientific applications); and (4) using the storage designer, array management techniques, and various replication and distribution strategies to deploy a software system with high performance on large data sets.ChunkyStore will substantially improve the performance of database systems and demonstrate that they can be used to manage a variety of scientific data. Education and dissemination efforts will be conducted via research papers, an open source release, several workshops for users, and integration with course projects in the graduate database systems class at MIT. For more information on ChunkyStore, including publications and source code, see http://db.csail.mit.edu/chunkystore/ .