Collaborative Research: Scalable I/O Middleware and File System Optimizations for High-Performance Computing

协作研究：高性能计算的可扩展 I/O 中间件和文件系统优化

• 批准号: 0621402
• 负责人: Mahmut Kandemir
• 金额: $ 14.95万
• 依托单位: Pennsylvania State Univ University Park
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2006
• 资助国家: 美国
• 起止时间: 2006-09-15 至 2010-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0621402&HistoricalAwards=false
• 关键词: Collaborative; Research; Scalable; Middleware; File

For data-intensive applications, I/O and storage layers are extremely critical, and are often overlooked, but they become a bottleneck in not only obtaining scalable performance but also in utilization and productivity of systems and application scientists. Along with computational capabilities, scalable software for I/O, and storage for the required capacity and performance must be developed in order to address the data intensive nature of applications and reap benefits in performance and productivity of High End Systems.This proposal entails research and development to address several parallel I/O problems in the HECURA initiative. In particular, the main goals of this proposal are to design and implement novel I/O middleware techniques and optimizations, parallel file system techniques that scale to ultra-scale systems, design and development of techniques that efficiently enable newer APIs including suggested extensions to POSIX for parallelism, and flexible I/O benchmarks that mimic real and dynamic I/O behavior of science and engineering applications. The PIs propose innovative techniques to optimize data accesses that utilize the understanding of high-level access patterns, and use that information through middleware and file systems to enable optimizations. Specifically, the objectives are to (1) design and develop middleware I/O optimizations and cache system that are able to capture small, unaligned, irregular I/O accesses from large number of processors and uses access pattern information to optimize for I/O; (2) incorporate these optimizations in MPICH2's MPI-IO implementation to make them available to a large number of users; (3) design scalable parallel file system techniques and optimizations including a versioning parallel file system, programmable and adaptable consistency semantics, layout optimizations, and self-tuning capabilities; (4) design and evaluate enhanced APIs for file system scalability, particularly for recently proposed enhancements to the POSIX interface (API) to enable highperformance parallel I/O; and (5) develop flexible, execution oriented and scalable I/O benchmarks that mimic the I/O behavior of real science, engineering and bioinformatics applications.

对于数据密集型应用程序，I/O和存储层极其关键，经常被忽视，但它们不仅成为获得可扩展性能的瓶颈，而且还成为系统和应用程序科学家的利用率和生产力的瓶颈。除了计算能力，还必须开发用于I/O的可扩展软件，以及用于所需容量和性能的存储，以解决应用程序的数据密集型特性，并在高端系统的性能和生产力方面获得好处。这项建议需要进行研究和开发，以解决HECURA计划中的几个并行I/O问题。具体地说，该提案的主要目标是设计和实现新颖的I/O中间件技术和优化、可扩展到超大规模系统的并行文件系统技术、设计和开发有效地启用更新的API的技术，包括建议的用于并行的POSIX扩展，以及模拟科学和工程应用的真实和动态I/O行为的灵活I/O基准。PI提出了优化数据访问的创新技术，这些技术利用对高级访问模式的理解，并通过中间件和文件系统使用该信息来实现优化。具体地说，目标是(1)设计和开发能够捕获来自大量处理器的小的、未对齐的、不规则的I/O访问并使用访问模式信息来针对I/O进行优化的中间件I/O优化和缓存系统；(2)将这些优化整合到MPICH2MPI-IO实现中，以使其可供大量用户使用；(3)设计可扩展的并行文件系统技术和优化，包括版本化并行文件系统、可编程和可适应的一致性语义、布局优化和自我调整能力；(4)设计和评估用于文件系统可伸缩性的增强API，特别是最近提出的对POSIX接口(API)的增强以实现高性能并行I/O；以及(5)开发灵活的、面向执行的和可扩展的I/O基准，以模拟真实科学、工程和生物信息学应用的I/O行为。