Collaborative Research: Peta-scale Storage for High End Computing

科学家和工程师在为数据密集型应用程序获得高性能I/O时所面临的困难因当前I/O系统呈现的低抽象级别而加剧。建议的研究将为开发人员创建更高级别的I/O抽象。建议的SSDS框架将I/O建模为I/O图，这些I/O图将应用程序组件与基于自主元数据构建的元数据的输入或输出机制(如文件系统)连接起来。可以对I/O图进行编程，以实现特定于应用程序的I/O功能，如数据过滤和转换、数据重新网格化以及类似任务。它们的管理是自动化的，包括将它们的逻辑图形节点映射到底层物理MPP和分布式机器资源。通过将I/O图的计算I/O操作与存储大容量数据的后端文件系统以及应用程序已经采取的I/O操作集成在一起，以及通过将元数据管理离线转移到元数据库中，SSDS中的I/O性能将得到改善。SSDS系统及其I/O图模型的实施将影响大量HPC用户社区，因为它计划与桑迪亚国家实验室(SNL)目前正在开发的轻量级文件系统(LWFS)集成。该文件系统及其SSDS扩展将部署在Sandia的大型机器上，以展示可伸缩性和应用程序实用程序。固态硬盘将与桑迪亚和橡树岭国家实验室(ORNL)的其他团队使用的文件格式和文件系统集成在一起

The difficulties faced by scientists and engineers in attaining high performance I/O for data-intensive applications are exacerbated by the low level of abstraction presented by current I/O systems. Proposed research will create higher level I/O abstractions for developers.The proposed SSDS framework models I/O as I/O Graphs that `connect' application componentswith input or output mechanisms like file systems based on metadata constructed offine by autonomousmetabots. I/O Graphs can be programmed to realize application-specific I/O functionality, such as data filtering and conversion, data remeshing, and similar tasks. Their management is automated, including themapping of their logical graph nodes to underlying physical MPP and distributed machine resources. I/Operformance in SSDS will be improved by integrating the computational I/O actions of I/O Graphs with thebackend file systems that store high volume data and with the I/O actions already taken by applications,and by moving metadata management offline into metabots.The implementation of the SSDS system and its I/O Graph model will impact a substantialHPC user community, due to its planned integration with the Lightweight File System (LWFS) currentlyunder development at Sandia National Laboratories (SNL). This file system and its SSDS extensions will bedeployed on large-scale machines at Sandia to demonstrate scalability and application utility. SSDS will beintegrated with the file formats and file systems used by other groups at Sandia and at Oak Ridge NationalLaboratories (ORNL)