HEC: Collaborative Research: SAM^2 Toolkit: Scalable and Adaptive Metadata Management for High-End Computing

高端计算应用对Exa字节级存储容量的需求不断增加，这要求比当前文件和存储系统提供的可扩展性和可靠性更高级别。 该建议涉及文件系统的研究，可扩展的基于集群的并行和分布式文件存储系统的元数据管理在HEC环境。它旨在开发一个可扩展和自适应元数据管理（SAM 2）工具包，以扩展高性能计算社区使用的最先进的基于集群的并行和分布式文件存储系统的功能，并充分利用其所承诺的峰值性能。有大量关于数据移动和管理扩展的研究，但是，对扩展基于集群的文件系统和I/O属性（即元数据）的需求被低估了。了解元数据流量的特点，采用适当的负载均衡、缓存、预取和分组机制来进行元数据管理，可以提高系统的可扩展性。可以预期，通过将可伸缩和自适应元数据管理组件适当地插入到现有技术的基于集群的并行和分布式文件存储系统中，可以潜在地提高应用和文件系统的性能，并且有助于将这种系统的高峰性能的承诺和潜力转化为真实的应用性能改进。 该项目包括以下组成部分：1.开发多变量预测模型来分析和预测文件元数据访问模式。2.开发可扩展和自适应的文件名映射方案，使用重复的布隆过滤器阵列技术来执行负载平衡并增加可扩展性3.开发去中心化的、位置感知的元数据分组方案，以促进批量元数据操作，如预取。4.使用分布式共享对象模型为客户端和服务器端元数据缓存开发自适应缓存一致性协议。5.将SAM 2组件原型化为最先进的并行虚拟文件系统PVFS 2和分布式存储数据缓存系统，为内布拉斯加大学林肯分校的DOE CMS Tier 2站点建立实验框架，并进行基准测试、评估和验证研究。

The increasing demand for Exa-byte-scale storage capacity by high end computing applications requires a higher level of scalability and dependability than that provided by current file and storage systems. The proposal deals with file systems research for metadata management of scalable cluster-based parallel and distributed file storage systems in the HEC environment. It aims to develop a scalable and adaptive metadata management (SAM2) toolkit to extend features of and fully leverage the peak performance promised by state-of-the-art cluster-based parallel and distributed file storage systems used by the high performance computing community. There is a large body of research on data movement and management scaling, however, the need to scale up the attributes of cluster-based file systems and I/O, that is, metadata, has been underestimated. An understanding of the characteristics of metadata traffic, and an application of proper load-balancing, caching, prefetching and grouping mechanisms to perform metadata management correspondingly, will lead to a high scalability. It is anticipated that by appropriately plugging the scalable and adaptive metadata management components into the state-of-the-art cluster-based parallel and distributed file storage systems one could potentially increase the performance of applications and file systems, and help translate the promise and potential of high peak performance of such systems to real application performance improvements. The project involves the following components: 1. Develop multi-variable forecasting models to analyze and predict file metadata access patterns. 2. Develop scalable and adaptive file name mapping schemes using the duplicative Bloom filter array technique to enforce load balance and increase scalability 3. Develop decentralized, locality-aware metadata grouping schemes to facilitate the bulk metadata operations such as prefetching. 4. Develop an adaptive cache coherence protocol using a distributed shared object model for client-side and server-side metadata caching. 5. Prototype the SAM2 components into the state-of-the-art parallel virtual file system PVFS2 and a distributed storage data caching system, set up an experimental framework for a DOE CMS Tier 2 site at University of Nebraska-Lincoln and conduct benchmark, evaluation and validation studies.