Hybrid OLTP/OLAP Main Memory Database System Based on Virtual Memory Snapshots

基于虚拟内存快照的混合OLTP/OLAP主存数据库系统

• 批准号: 194871756
• 负责人: Professor Dr. Thomas Neumann
• 金额: --
• 依托单位: Lehrstuhl für Informatik III: Datenbanksysteme
• 依托单位国家: 德国
• 项目类别: Research Grants
• 财政年份: 2011
• 资助国家: 德国
• 起止时间: 2010-12-31 至 2018-12-31
• 项目状态: 已结题
• 来源: https://gepris.dfg.de/gepris/projekt/194871756?language=en
• 关键词: Hybrid; OLTP; OLAP; Main; Memory

In the first phase of this project we developped a hybrid main memory database system that supports OLTP- and OLAP-applications. The key idea is to exploit the OS/processor-support for virtual memory management. This allows to spawn consistent database snapshots to isolate OLAP queries from the OLTP transactions -- even though they share the same database. Based on the first prototype, called HyPer, that existed already at the start of the project and proved the feasibility of such a hybrid OLTP&OLAP database, we realized in the course of the project a full-fledged database system. The numerous publications in the internationally most prestigious venues (SIGMOD, ICDE, CIDR, and VLDB) suggest that HyPer is one of the most visible academic database projects. This claim is also supported by the Best Paper Award of the ICDE 2014as well as the prestigious VLDB Early Career Award 2014 for T. Neumann's work on the HyPer database engine. In the continuation period we want to focus on the following topics:1) Multi-version concurrency control schemes are a promising method for synchronizing OLTP (online transaction processing) transactions without much overhead for read-only transactions. Furthermore, the previous work on hardware transactional memory should be extended for synchronizing the new adaptive radix-tree indexing.2) So far we have concentrated on optimizing the performance of SQL 92 queries. For decision support queries window functions constitute an important extension of the query language. So far no prior research can be found on massively parallel evalaution of queries containing window functions which is, however, essential to exploit the vast aggregated compute power of modern many-core servers.3) In the near future servers will consist of heterogeneous components (GPUs, energy-safing CPUs, high-performance CPUs, etc.) In order to utilize these components in an energy-efficient and/or performance-optimized way a scheduler is needed that allocates query evaluation tasks to the best-suited hardware component. 4) The hardware vendors (Intel, HP, etc.) are currently developing non-volatile DRAM. This new storage technology is particularly suitable for main-memory database systems like HyPer to reduce fail-over times during recovery and thus increase the availability. Each of these topics is sufficiently complex for a PhD dissertation topic. The existing HyPer implementation will be used as an experimental platform for all engineering work in order to prove its practical relevance.

在这个项目的第一阶段，我们开发了一个支持OLTP和olap应用程序的混合主存数据库系统。关键思想是利用操作系统/处理器对虚拟内存管理的支持。这允许生成一致的数据库快照，将OLAP查询与OLTP事务隔离开来——即使它们共享相同的数据库。基于项目开始时已经存在的第一个原型HyPer，并证明了这种混合OLTP&OLAP数据库的可行性，我们在项目过程中实现了一个完整的数据库系统。在国际上最负盛名的场所（SIGMOD， ICDE， CIDR和VLDB）的众多出版物表明HyPer是最引人注目的学术数据库项目之一。这一说法也得到了ICDE 2014年最佳论文奖的支持，以及著名的VLDB 2014年早期职业奖，以表彰T. Neumann在HyPer数据库引擎上的工作。在延续期间，我们希望关注以下主题：1)多版本并发控制方案是一种很有前途的方法，用于同步OLTP（在线事务处理）事务，而不会对只读事务产生太多开销。此外，应该扩展先前在硬件事务内存方面的工作，以同步新的自适应基数树索引。2)到目前为止，我们专注于优化SQL 92查询的性能。对于决策支持查询，窗口函数是查询语言的重要扩展。到目前为止，还没有关于包含窗口函数的查询的大规模并行计算的研究，然而，这对于利用现代多核服务器的巨大聚合计算能力是必不可少的。3)在不久的将来，服务器将由异构组件（gpu，节能cpu，高性能cpu等）组成，为了以节能和/或性能优化的方式利用这些组件，需要一个调度程序来分配查询评估任务到最适合的硬件组件。4)硬件厂商（英特尔、惠普等）目前正在开发非易失性DRAM。这种新的存储技术特别适用于像HyPer这样的主存数据库系统，可以减少恢复期间的故障转移时间，从而提高可用性。这些主题中的每一个都足够复杂，可以作为博士论文的主题。现有的HyPer实现将被用作所有工程工作的实验平台，以证明其实际意义。

项目成果

Morsel-driven parallelism: a NUMA-aware query evaluation framework for the many-core age

• DOI: 10.1145/2588555.2610507
• 发表时间: 2014-06
• 期刊: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data
• 作者: Viktor Leis;P. Boncz;A. Kemper;Thomas Neumann

High-Speed Query Processing over High-Speed Networks

• DOI: 10.14778/2856318.2856319
• 发表时间: 2015-02
• 期刊: ArXiv
• 作者: Wolf Rödiger;Tobias Mühlbauer;A. Kemper;Thomas Neumann

Fast Serializable Multi-Version Concurrency Control for Main-Memory Database Systems

• DOI: 10.1145/2723372.2749436
• 发表时间: 2015-05
• 期刊: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data
• 作者: Thomas Neumann;Tobias Mühlbauer;A. Kemper

Data Blocks: Hybrid OLTP and OLAP on Compressed Storage using both Vectorization and Compilation

• DOI: 10.1145/2882903.2882925
• 发表时间: 2016-06
• 期刊: Proceedings of the 2016 International Conference on Management of Data
• 作者: Harald Lang;Tobias Mühlbauer;Florian Funke;P. Boncz;Thomas Neumann;A. Kemper

HyPer: A hybrid OLTP&OLAP main memory database system based on virtual memory snapshots

HyPer：混合 OLTP

• DOI: 10.1109/icde.2011.5767867
• 发表时间: 2011
• 期刊: 2011 IEEE 27th International Conference on Data Engineering
• 作者: A. Kemper;T. Neumann
• 通讯作者: T. Neumann