III-COR - ChunkyStore: Physical Database Design for Next-Generation Databases

ChunkyStore：下一代数据库的物理数据库设计传统数据库在磁盘上以行为主的格式布局表，但最近的研究表明，其他物理布局（例如，使用以列为主的格式）在许多应用中可以产生显著的（数量级）性能增益。ChunkyStore项目的主要目标是探索物理存储布局和数据库性能之间的关系。 实现这一目标的主要机制是将数据库表划分为具有不同数量的同位行和列的“块”，并随着时间的推移自适应地调整这些块以优化性能。实现这一目标的具体技术包括：（1）研究不同“块”算法在不同工作负载和应用（包括科学数据管理和数据仓库）下的性能;（2）构建自动存储设计器，该设计器可根据历史工作负载模式的模型选择最佳的表分块;（3）研究非表格数据（如影像阵列）的基于分块的表示和布局（在科学应用中尤为重要）;以及（4）使用存储设计器、阵列管理技术以及各种复制和分发策略，以在大型数据集上部署高性能的软件系统。ChunkyStore将大大提高数据库系统的性能，并证明它们可用于管理各种科学数据。 教育和传播工作将通过研究论文，开源版本，用户的几个研讨会，以及与麻省理工学院研究生数据库系统课程的课程项目整合进行。 有关ChunkyStore的更多信息，包括出版物和源代码，请参见http://db.csail.mit.edu/chunkystore/。

ChunkyStore: Physical Database Design for Next-Generation DatabasesTraditional databases lay out tables on disk in a row-major format, but recent research has shown that other physical layouts (e.g., using a column-major format) can yield substantial (order-of-magnitude) performance gains in many applications. The broad goal of the ChunkyStore project is to explore this relationship between physical storage layouts and database performance. The primary mechanism for doing this is to partition database tables into "chunks" of varying numbers of co-located rows and columns and adaptively adjusting these chunks over time to optimize performance.Specific techniques used to achieve this goal include: (1) investigation of the performance of different "chunking" algorithms under different workloads and applications, including scientific data management and data warehousing; (2) building an automatic storage designer that chooses the best chunking of tables based on models of historical workload patterns; (3) studying chunk-based representations and layout of non-tabular data, such as arrays of imagery (which are particularly important in scientific applications); and (4) using the storage designer, array management techniques, and various replication and distribution strategies to deploy a software system with high performance on large data sets.ChunkyStore will substantially improve the performance of database systems and demonstrate that they can be used to manage a variety of scientific data. Education and dissemination efforts will be conducted via research papers, an open source release, several workshops for users, and integration with course projects in the graduate database systems class at MIT. For more information on ChunkyStore, including publications and source code, see http://db.csail.mit.edu/chunkystore/ .