Coupled Storage System for Efficient Management of Self-Describing Data Formats

用于高效管理自描述数据格式的耦合存储系统

• 批准号: 417705296
• 负责人: Professor Dr. Michael Kuhn
• 金额: --
• 依托单位: Deutsches Klimarechenzentrum GmbH
• 依托单位国家: 德国
• 项目类别: Research Grants
• 财政年份: 2019
• 资助国家: 德国
• 起止时间: 2018-12-31 至 2023-12-31
• 项目状态: 已结题
• 来源: https://gepris.dfg.de/gepris/projekt/417705296?language=en
• 关键词: Coupled; Storage; System; Efficient; Management

Over the last decades, societies came to rely more than ever on technological progress in information technology. Especially in the area of scientific research, this does enable the possibility to solve increasingly complex problems, which nowadays require the computational power of supercomputers. The rising complexity of the processed problems as well as the growth of computation power leads to rapidly increasing data volumes; the globally produced data volume doubles approximately every two years, leading to an exponential data deluge. This imposes a serious problem as the development of the storage and network technologies is considerably slower. The result is a widening gap between the performance of computing and storage devices, resulting in a storage bottleneck. This is especially true for large-scale systems found in high-performance computing. To ease this situation, a hierarchy of different storage devices is used to suffice the demand for high capacity on the one hand and for high velocity as well as reliability on the other hand. By combining the advantages of different storage technologies, the overall performance is significantly increased while inducing lesser costs for acquisition, operation and maintenance. However, for future exascale systems, the difficulties will get even worse, requiring critical improvements in order to exploit the systems' capabilities. The existing input/output (I/O) stack leads to additional performance and management issues.The produced data is typically stored using self-describing data formats to facilitate exchange and analysis within the scientific community. The project goal is to explore the benefits of a coupled storage system for these formats. It will introduce a novel hybrid approach leveraging storage technologies from the fields of high-performance computing and database systems, where each technology will be used according to its respective strengths and weaknesses. By coupling the storage system tightly with self-describing data formats, it can make use of structural information for selecting appropriate storage technologies and tiers. As such information is currently not available, storage systems have to employ heuristics, which often lead to suboptimal performance as well as unnecessary and expensive data movements. Moreover, the storage system will support adaptable I/O semantics to tune its performance according to application and data format requirements. Together, these features will enable completely new data management methods and provide significant performance improvements. Existing workflows of scientific users will be supported through a dedicated data analysis interface. All changes will be thoroughly tested to ensure backwards compatibility with existing applications and interfaces. Consequently, no modifications will be necessary to run applications on top of CoSEMoS, which helps preserve past investments in scientific software development.

在过去几十年中，社会比以往任何时候都更加依赖信息技术的技术进步。特别是在科学研究领域，这使得解决日益复杂的问题成为可能，这些问题现在需要超级计算机的计算能力。所处理问题的复杂性不断增加，计算能力也不断提高，导致数据量迅速增加;全球产生的数据量大约每两年翻一番，导致数据呈指数级泛滥。这带来了一个严重的问题，因为存储和网络技术的发展相当缓慢。结果是计算设备和存储设备的性能之间的差距越来越大，从而导致存储瓶颈。对于高性能计算中的大规模系统尤其如此。为了缓解这种情况，使用不同存储设备的层次结构来满足一方面对高容量的需求，另一方面对高速度和可靠性的需求。通过结合不同存储技术的优势，整体性能显著提高，同时降低了采购、操作和维护成本。然而，对于未来的艾级系统，困难将变得更糟，需要关键的改进才能发挥系统的能力。现有的输入/输出（I/O）堆栈导致额外的性能和管理问题。生成的数据通常使用自描述数据格式存储，以促进科学界内部的交换和分析。该项目的目标是探索这些格式的耦合存储系统的好处。它将引入一种新的混合方法，利用高性能计算和数据库系统领域的存储技术，每种技术将根据其各自的优势和劣势加以使用。通过将存储系统与自描述数据格式紧密耦合，它可以利用结构信息来选择适当的存储技术和层。由于这些信息目前不可用，因此存储系统必须采用并行计算，这通常导致次优性能以及不必要和昂贵的数据移动。此外，存储系统将支持自适应I/O语义，以根据应用程序和数据格式要求调整其性能。这些功能将共同实现全新的数据管理方法，并提供显着的性能改进。科学用户的现有工作流程将通过专用数据分析界面得到支持。所有更改都将经过彻底测试，以确保与现有应用程序和接口的向后兼容性。因此，在CoSEMoS上运行应用程序不需要进行任何修改，这有助于保留过去在科学软件开发方面的投资。