Collaborative Research: Scalable I/O Middleware and File System Optimizations for High-Performance Computing

对于数据密集型应用程序，I/O和存储层非常关键，而且经常被忽视，但它们不仅成为获得可伸缩性能的瓶颈，而且还成为系统和应用程序科学家的利用率和生产力的瓶颈。除了计算能力之外，还必须开发用于I/O的可扩展软件，以及用于所需容量和性能的存储，以解决应用程序的数据密集型特性，并从高端系统的性能和生产力方面获益。该提案需要研究和开发，以解决HECURA倡议中的几个并行I/O问题。特别是，该提案的主要目标是设计和实现新颖的I/O中间件技术和优化，可扩展到超大规模系统的并行文件系统技术，设计和开发有效启用新api的技术，包括建议的POSIX并行扩展，以及模拟科学和工程应用的真实和动态I/O行为的灵活I/O基准。pi提出了利用对高级访问模式的理解来优化数据访问的创新技术，并通过中间件和文件系统使用该信息来实现优化。具体来说，目标是：(1)设计和开发中间件I/O优化和缓存系统，这些系统能够捕获来自大量处理器的小的、未对齐的、不规则的I/O访问，并使用访问模式信息来优化I/O；(2)将这些优化整合到MPICH2的MPI-IO实现中，使其可供大量用户使用；(3)设计可扩展的并行文件系统技术和优化，包括版本控制并行文件系统、可编程和可适应的一致性语义、布局优化和自调优能力；(4)设计和评估用于文件系统可扩展性的增强API，特别是最近提出的对POSIX接口（API）的增强，以实现高性能并行I/O；(5)开发灵活的、面向执行的、可扩展的I/O基准，模拟真实科学、工程和生物信息学应用的I/O行为。

For data-intensive applications, I/O and storage layers are extremely critical, and are often overlooked, but they become a bottleneck in not only obtaining scalable performance but also in utilization and productivity of systems and application scientists. Along with computational capabilities, scalable software for I/O, and storage for the required capacity and performance must be developed in order to address the data intensive nature of applications and reap benefits in performance and productivity of High End Systems.This proposal entails research and development to address several parallel I/O problems in the HECURA initiative. In particular, the main goals of this proposal are to design and implement novel I/O middleware techniques and optimizations, parallel file system techniques that scale to ultra-scale systems, design and development of techniques that efficiently enable newer APIs including suggested extensions to POSIX for parallelism, and flexible I/O benchmarks that mimic real and dynamic I/O behavior of science and engineering applications. The PIs propose innovative techniques to optimize data accesses that utilize the understanding of high-level access patterns, and use that information through middleware and file systems to enable optimizations. Specifically, the objectives are to (1) design and develop middleware I/O optimizations and cache system that are able to capture small, unaligned, irregular I/O accesses from large number of processors and uses access pattern information to optimize for I/O; (2) incorporate these optimizations in MPICH2's MPI-IO implementation to make them available to a large number of users; (3) design scalable parallel file system techniques and optimizations including a versioning parallel file system, programmable and adaptable consistency semantics, layout optimizations, and self-tuning capabilities; (4) design and evaluate enhanced APIs for file system scalability, particularly for recently proposed enhancements to the POSIX interface (API) to enable highperformance parallel I/O; and (5) develop flexible, execution oriented and scalable I/O benchmarks that mimic the I/O behavior of real science, engineering and bioinformatics applications.