Collaborative Research: Peta-scale Storage for High End Computing

合作研究：用于高端计算的千万亿级存储

• 批准号: 0621393
• 负责人: Karsten Schwan
• 金额: --
• 依托单位: Georgia Tech Research Corporation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2006
• 资助国家: 美国
• 起止时间: 2006-09-01 至 2010-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0621393&HistoricalAwards=false
• 关键词: Collaborative; Research; Peta; scale; Storage

The difficulties faced by scientists and engineers in attaining high performance I/O for data-intensive applications are exacerbated by the low level of abstraction presented by current I/O systems. Proposed research will create higher level I/O abstractions for developers.The proposed SSDS framework models I/O as I/O Graphs that `connect' application componentswith input or output mechanisms like file systems based on metadata constructed offine by autonomousmetabots. I/O Graphs can be programmed to realize application-specific I/O functionality, such as data filtering and conversion, data remeshing, and similar tasks. Their management is automated, including themapping of their logical graph nodes to underlying physical MPP and distributed machine resources. I/Operformance in SSDS will be improved by integrating the computational I/O actions of I/O Graphs with thebackend file systems that store high volume data and with the I/O actions already taken by applications,and by moving metadata management offline into metabots.The implementation of the SSDS system and its I/O Graph model will impact a substantialHPC user community, due to its planned integration with the Lightweight File System (LWFS) currentlyunder development at Sandia National Laboratories (SNL). This file system and its SSDS extensions will bedeployed on large-scale machines at Sandia to demonstrate scalability and application utility. SSDS will beintegrated with the file formats and file systems used by other groups at Sandia and at Oak Ridge NationalLaboratories (ORNL)

科学家和工程师在为数据密集型应用程序实现高性能I/O方面所面临的困难由于当前I/O系统所呈现的低抽象级别而加剧。建议的研究将为开发人员创建更高级别的I/O抽象。建议的SSDS框架将I/O建模为I/O图，该I/O图将应用程序组件与输入或输出机制（如基于元数据的文件系统）“连接”，这些元数据由非正式元数据构建。可以对I/O Graph进行编程，以实现特定于应用程序的I/O功能，例如数据过滤和转换、数据重新网格化以及类似的任务。它们的管理是自动化的，包括将它们的逻辑图节点映射到底层物理MPP和分布式机器资源。SSDS中的I/O性能将通过将I/O Graph的计算I/O操作与存储大量数据的后端文件系统以及应用程序已经采取的I/O操作集成，并通过将元数据管理离线移动到元数据中来提高。SSDS系统及其I/O Graph模型的实现将影响大量HPC用户社区，这是因为它计划与Sandia国家实验室（SNL）目前正在开发的轻量级文件系统（LWFS）集成。该文件系统及其SSDS扩展将部署在Sandia的大规模机器上，以展示可扩展性和应用程序实用性。SSDS将与桑迪亚和橡树岭国家实验室（ORNL）的其他小组使用的文件格式和文件系统集成