III-COR - ChunkyStore: Physical Database Design for Next-Generation Databases

ChunkyStore：下一代数据库的物理数据库设计传统数据库以行为主的格式在磁盘上布局表，但最近的研究表明，在许多应用程序中，其他物理布局(例如，使用列为主的格式)可以带来显著的(数量级)性能提升。ChunkyStore项目的主要目标是探索物理存储布局和数据库性能之间的这种关系。用于实现这一目标的具体技术包括：(1)调查不同工作负荷和应用程序下不同“分块”算法的性能，包括科学数据管理和数据仓库；(2)建立基于历史工作负荷模式模型的自动存储设计器；(3)研究基于块的非表格数据的表示和布局，例如图像阵列(这在科学应用中特别重要)；使用存储设计器、阵列管理技术以及各种复制和分发策略，在大数据集上部署高性能的软件系统。ChunkyStore将大幅提高数据库系统的性能，并证明它们可以用于管理各种科学数据。教育和传播工作将通过研究论文、开放源码发布、几个用户研讨会以及与麻省理工学院研究生数据库系统课程项目的整合来进行。有关ChunkyStore的更多信息，包括出版物和源代码，请参阅http://db.csail.mit.edu/chunkystore/。

ChunkyStore: Physical Database Design for Next-Generation DatabasesTraditional databases lay out tables on disk in a row-major format, but recent research has shown that other physical layouts (e.g., using a column-major format) can yield substantial (order-of-magnitude) performance gains in many applications. The broad goal of the ChunkyStore project is to explore this relationship between physical storage layouts and database performance. The primary mechanism for doing this is to partition database tables into "chunks" of varying numbers of co-located rows and columns and adaptively adjusting these chunks over time to optimize performance.Specific techniques used to achieve this goal include: (1) investigation of the performance of different "chunking" algorithms under different workloads and applications, including scientific data management and data warehousing; (2) building an automatic storage designer that chooses the best chunking of tables based on models of historical workload patterns; (3) studying chunk-based representations and layout of non-tabular data, such as arrays of imagery (which are particularly important in scientific applications); and (4) using the storage designer, array management techniques, and various replication and distribution strategies to deploy a software system with high performance on large data sets.ChunkyStore will substantially improve the performance of database systems and demonstrate that they can be used to manage a variety of scientific data. Education and dissemination efforts will be conducted via research papers, an open source release, several workshops for users, and integration with course projects in the graduate database systems class at MIT. For more information on ChunkyStore, including publications and source code, see http://db.csail.mit.edu/chunkystore/ .