Scaling Beyond DRAM with PMem without Compromising Performance

在过去的十年中，我们的团队专注于开发内存数据库技术，最著名的是HyPer数据库。在过去二十年中，主存储器容量的增加使得内存数据处理成为可能，同时提供了前所未有的性能。尽管大多数事务性数据适合最大的扩展服务器，可以提供几TB的DRAM容量，但我们应该重新考虑永久内存（PMem）的新机会，原因有两个：(1)与纯DRAM系统相比，它的成本/性能比似乎更有利；(2)大数据时代的新应用需要更高的数据量。其他开发则试图通过向外扩展解决方案来解决后者。然而，它们会产生更高的通信开销——即使使用新的昂贵的通信技术（如InfiniBand和RDMA）也是如此。作为替代方案，我们希望通过利用扩展服务器中新兴的PMem功能来扩展DRAM容量之外的容量，而不影响纯内存中的处理性能（对于那些适合大容量DRAM和PMem的工作集）。超过DRAM + PMem容量的工作集应该只会通过依赖快速SSD存储而导致适当的降级。这里提出的研究工作将整合到Umbra数据库系统中。Umbra是我们成功的纯内存系统HyPer的“精神”继承者和进化。它完全消除了HyPer对数据放入主存的限制。Umbra是一个长期的研究项目，有很多大数据应用。在这个为期三年的项目中，我们专注于如何将持久存储器等现代存储系统集成为Umbra的一流公民。这需要重新设计数据的存储和索引结构，以及缓冲、日志记录和恢复等关键功能组件。我们想要实现的总体目标是，如果工作集适合DRAM + PMem，则扩展到昂贵的DRAM容量之外，而不会减慢速度；如果扩展到DRAM + PMem，则会优雅地降低性能。完整的数据库系统原型Umbra作为一个集成平台，用于研究将持久内存集成到数据库系统中的各种可能性。因此，Umbra构成了一个现实的（和无价的）测试平台，用于端到端的调查，以实现和证明基础研究的实际相关性。在Jana Giceva教授加入TUM小组之后，pi小组（几乎是独一无二的）涵盖数据库和操作系统经验的广泛专业知识将促进最有效地利用现代存储设备-无论是在DBMS还是在操作系统层面上“更深”和更通用。

Over the last decade our group has focused on developing in-memory database technology, most notably the HyPer database. The increases in main memory capacities over the last two decades have made in-memory data processing feasible while offering unprecedented performance. Even though most transactional data fits into the largest scale-up servers that facilitate several TB of DRAM capacity, we should reconsider the new opportunities of persistent memory (PMem) for two reasons: (1) its cost/performance ratio appears beneficial in comparison to pure DRAM systems and (2) new applications in the Big Data era require even higher data volumes. Other developments try to tackle the latter with scale-out solutions. However, they incur a much higher communication overhead – even when using new costly communication technologies like InfiniBand and RDMA. As an alternative, we want to scale beyond DRAM capacity by exploiting emerging PMem capabilities in scale-up servers without compromising the pure in-memory processing performance (for those working sets that fit into the large capacity DRAM and PMem). Working sets beyond DRAM plus PMem capacity should induce only a graceful degradation by relying on fast SSD storage. The research work proposed here shall be integrated into the Umbra database system. Umbra is the "spiritual" successor and evolution of our successful pure in-memory system HyPer. It completely eliminates the restrictions of HyPer that data fits into main memory. Umbra is a long-term research project with many Big Data applications. In this three-year project, we concentrate on how modern storage systems such as persistent memory can be integrated as a first-class citizen of Umbra. This requires to re-design the storage- and index-structures for data as well as key functional components like buffering, logging and recovery. The overall goal that we want to achieve is scaling beyond costly DRAM capacity without slowing down if the working set fits into DRAM plus PMem and gracefully degrading if it grows beyond. The full-fledged database system prototype Umbra serves as an integration platform for researching the spectrum of possibilities for the incorporation of persistent memory into a database system. Thereby, Umbra constitutes a realistic (and invaluable) test bed for an end-to-end investigation to achieve and prove practical relevance of the foundational research. The (almost unique) broad expertise of the PIs’ groups covering database as well as OS experience (after Prof. Jana Giceva joined the TUM group) will facilitate the exploitation of the modern storage devices most effectively – either at DBMS or "deeper" and more generic at the OS level.