Collaborative Research: Peta-scale Storage for High End Computing

科学家和工程师在为数据密集型应用程序获得高性能I/O时所面临的困难，由于当前I/O系统所呈现的低抽象水平而加剧。拟议的研究将为开发人员创建更高级别的I/O抽象。提议的SSDS框架将I/O建模为I/O图，它将应用程序组件与输入或输出机制（如基于自主元机器人离线构建的元数据的文件系统）“连接”起来。可以对I/O图进行编程，以实现特定于应用程序的I/O功能，例如数据过滤和转换、数据重划分以及类似的任务。它们的管理是自动化的，包括将它们的逻辑图节点映射到底层物理MPP和分布式机器资源。通过将I/O图的计算I/O操作与存储大量数据的后端文件系统和应用程序已经采取的I/O操作集成，以及将元数据管理离线转移到元机器人中，ssd中的I/O性能将得到改善。SSDS系统及其I/O图形模型的实施将影响大量hpc用户社区，因为它计划与桑迪亚国家实验室（SNL）正在开发的轻量级文件系统（LWFS）集成。该文件系统及其ssd扩展将部署在桑迪亚的大型机器上，以演示可伸缩性和应用程序的实用性。ssd将与桑迪亚和橡树岭国家实验室（ORNL）的其他小组使用的文件格式和文件系统集成。

The difficulties faced by scientists and engineers in attaining high performance I/O for data-intensive applications are exacerbated by the low level of abstraction presented by current I/O systems. Proposed research will create higher level I/O abstractions for developers.The proposed SSDS framework models I/O as I/O Graphs that `connect' application componentswith input or output mechanisms like file systems based on metadata constructed offine by autonomousmetabots. I/O Graphs can be programmed to realize application-specific I/O functionality, such as data filtering and conversion, data remeshing, and similar tasks. Their management is automated, including themapping of their logical graph nodes to underlying physical MPP and distributed machine resources. I/Operformance in SSDS will be improved by integrating the computational I/O actions of I/O Graphs with thebackend file systems that store high volume data and with the I/O actions already taken by applications,and by moving metadata management offline into metabots.The implementation of the SSDS system and its I/O Graph model will impact a substantialHPC user community, due to its planned integration with the Lightweight File System (LWFS) currentlyunder development at Sandia National Laboratories (SNL). This file system and its SSDS extensions will bedeployed on large-scale machines at Sandia to demonstrate scalability and application utility. SSDS will beintegrated with the file formats and file systems used by other groups at Sandia and at Oak Ridge NationalLaboratories (ORNL)