A Common Storage Engine for Modern Memory and Storage Hierarchies

现代内存和存储层次结构的通用存储引擎

• 批准号: 502268500
• 负责人: Professor Dr. Michael Kuhn
• 金额: --
• 依托单位: Arbeitsgruppe Datenbanken und Software Engineering
• 依托单位国家: 德国
• 项目类别: Priority Programmes
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/502268500?language=en
• 关键词: Common; Storage; Engine; Modern; Memory

Scientific research is increasingly driven by data-intensive problems. As the complexity of studied problems is rising, so does their need for high data throughput and capacity. The globally produced data volume doubles approximately every two years, leading to an exponential data deluge. This deluge then directly challenges database management systems and file systems, which provide the foundation for efficient data analysis and management. These systems use different memory and storage devices, which were traditionally divided into primary, secondary and tertiary memory. However, with the introduction of the disruptive technology of non-volatile RAM (NVRAM), these classes started to merge into one another leading to heterogeneous storage architectures, where each storage device has highly different performance characteristics (e.g., persistence, storage capacity, latency). Hence, a major challenge is how to exploit the specific characteristics of memory devices.To this end, SMASH will investigate the benefits of a common storage engine that manages a heterogeneous storage landscape, including traditional storage devices and non-volatile memory technologies. The core for this storage engine will be B-epsilon-trees, as they can be used to efficiently exploit these different devices. Furthermore, data placement and migration strategies will be investigated to minimize the overhead caused by transferring data between different devices. Eliminating the need for volatile caches will allow data consistency guarantees to be improved. From the application side, the storage engine will offer key-value and object interfaces that can be used for a wide range of use cases, such as high-performance computing (HPC) and database management systems. Moreover, due to the widening gap between the performance of computing and storage devices as well as their stagnating access performance, data reduction techniques are in high demand to reduce the bandwidth requirements when storing and retrieving data. We will, therefore, conduct research regarding data transformations in general and the possibilities of external and accelerated transformations. As part of SMASH, we will provide a prototypical standalone software library to be used by third-party projects. Common HPC workflows will be supported through an integration of SMASH into the existing JULEA storage framework, while database systems can use the interface of SMASH directly whenever data is stored or accessed.The consortium will be supported by Intel Corporation, especially concerning the hardware acceleration of data transformations and collaborate actively with other project within the SPP 2377.

科学研究越来越受到数据密集型问题的驱动。随着研究问题的复杂性不断上升，对高数据吞吐量和容量的需求也在不断增加。全球产生的数据量大约每两年翻一番，导致指数级数据泛滥。这种泛滥直接挑战了数据库管理系统和文件系统，这些系统为有效的数据分析和管理提供了基础。这些系统使用不同的存储器和存储设备，传统上分为主存储器、次存储器和第三存储器。然而，随着非易失性RAM（NVRAM）的破坏性技术的引入，这些类别开始彼此合并，导致异构存储架构，其中每个存储设备具有高度不同的性能特性（例如，持久性、存储容量、延迟）。因此，一个主要的挑战是如何利用存储设备的特定特性。为此，SMASH将研究管理异构存储环境（包括传统存储设备和非易失性存储技术）的通用存储引擎的好处。这个存储引擎的核心将是B-Tree树，因为它们可以用来有效地利用这些不同的设备。此外，将研究数据放置和迁移策略，以最大限度地减少在不同设备之间传输数据所造成的开销。消除对易失性缓存的需求将允许提高数据一致性保证。从应用程序方面来看，存储引擎将提供键值和对象接口，可用于各种用例，例如高性能计算（HPC）和数据库管理系统。此外，由于计算设备和存储设备的性能之间的差距扩大以及它们的访问性能停滞不前，因此迫切需要数据缩减技术来降低存储和检索数据时的带宽要求。因此，我们将对一般数据转换以及外部和加速转换的可能性进行研究。作为SMASH的一部分，我们将提供一个原型独立软件库，供第三方项目使用。通过将SMASH集成到现有的JULEA存储框架中，将支持通用HPC工作流程，而数据库系统可以在存储或访问数据时直接使用SMASH接口。该联盟将得到英特尔公司的支持，特别是在数据转换的硬件加速方面，并与SPP 2377中的其他项目积极合作。