HEC: Collaborative Research: SAM^2 Toolkit: Scalable and Adaptive Metadata Management for High-End Computing

HEC：协作研究：SAM^2 工具包：用于高端计算的可扩展和自适应元数据管理

• 批准号: 0621526
• 负责人: Hong Jiang
• 金额: --
• 依托单位: University of Nebraska-Lincoln
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2006
• 资助国家: 美国
• 起止时间: 2006-08-15 至 2010-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0621526&HistoricalAwards=false
• 关键词: HEC; Collaborative; Research; SAM; Toolkit

The increasing demand for Exa-byte-scale storage capacity by high end computing applications requires ahigher level of scalability and dependability than that provided by current file and storage systems.The proposal deals with file systems research for metadata management of scalable cluster-based parallel and distributed file storage systems in the HEC environment. It aims to develop a scalable and adaptive metadata management (SAM2) toolkit to extend features of and fully leverage the peak performance promised by state-of-the-art cluster-based parallel and distributed file storage systems used by the high performance computing community. There is a large body of research on data movement and management scaling, however, the need to scale up the attributes of cluster-based file systems and I/O, that is, metadata, has been underestimated. An understanding of the characteristics of metadata traffic, and an application of proper load-balancing, caching, prefetching and grouping mechanisms to perform metadata management correspondingly, will lead to a high scalability. It is anticipated that by appropriately plugging the scalable and adaptive metadata management components into the state-of-the-art cluster-based parallel and distributed file storage systems one could potentially increase the performance of applications and file systems, and help translate the promise and potential of high peak performance of such systems to realapplication performance improvements.The project involves the following components:1. Develop multi-variable forecasting models to analyze and predict file metadata access patterns.2. Develop scalable and adaptive file name mapping schemes using the duplicative Bloom filter array technique to enforce load balance and increase scalability3. Develop decentralized, locality-aware metadata grouping schemes to facilitate the bulk metadata operations such as prefetching.4. Develop an adaptive cache coherence protocol using a distributed shared object model for client-side and server-side metadata caching.5. Prototype the SAM2 components into the state-of-the-art parallel virtual file system PVFS2 and a distributed storage data caching system, set up an experimental framework for a DOE CMS Tier 2 site at University of Nebraska-Lincoln and conduct benchmark, evaluation and validation studies.

随着高端计算应用对exa字节级存储容量的需求不断增长，对现有文件和存储系统的可扩展性和可靠性提出了更高要求。本文研究了HEC环境下基于可扩展集群的并行和分布式文件存储系统的元数据管理。它旨在开发一个可伸缩和自适应的元数据管理（SAM2）工具包，以扩展高性能计算社区使用的最先进的基于集群的并行和分布式文件存储系统的特性，并充分利用它们所承诺的峰值性能。有大量关于数据移动和管理扩展的研究，但是，扩展基于集群的文件系统和I/O属性（即元数据）的需求被低估了。了解元数据流量的特征，并应用适当的负载平衡、缓存、预取和分组机制来执行相应的元数据管理，将导致高可伸缩性。预计通过适当地将可伸缩和自适应元数据管理组件插入到最先进的基于集群的并行和分布式文件存储系统中，可以潜在地提高应用程序和文件系统的性能，并帮助将此类系统的高峰性能的承诺和潜力转化为实际应用程序性能改进。本项目包括以下几个部分：1。建立多变量预测模型，分析和预测文件元数据访问模式。使用重复Bloom过滤器阵列技术开发可扩展和自适应的文件名映射方案，以强制负载平衡并提高可伸缩性3。开发分散的、位置感知的元数据分组方案，以方便预取等批量元数据操作。使用分布式共享对象模型为客户端和服务器端元数据缓存开发自适应缓存一致性协议。将SAM2组件原型化到最先进的并行虚拟文件系统PVFS2和分布式存储数据缓存系统中，为内布拉斯加大学林肯分校的DOE CMS Tier 2站点建立实验框架，并进行基准测试、评估和验证研究。