Collaborative Research: Scalable I/O Middleware and File System Optimizations for High-Performance Computing

协作研究：高性能计算的可扩展 I/O 中间件和文件系统优化

• 批准号: 0621443
• 负责人: Alok Choudhary
• 金额: $ 52万
• 依托单位: Northwestern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2006
• 资助国家: 美国
• 起止时间: 2006-09-15 至 2012-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0621443&HistoricalAwards=false
• 关键词: Collaborative; Research; Scalable; Middleware; File

For data-intensive applications, I/O and storage layers are extremely critical, and are often overlooked, but they become a bottleneck in not only obtaining scalable performance but also in utilization and productivity of systems and application scientists. Along with computational capabilities, scalable software for I/O, and storage for the required capacity and performance must be developed in order to address the data intensive nature of applications and reap benefits in performance and productivity of High End Systems.This proposal entails research and development to address several parallel I/O problems in the HECURA initiative. In particular, the main goals of this proposal are to design and implement novel I/O middleware techniques and optimizations, parallel file system techniques that scale to ultra-scale systems, design and development of techniques that efficiently enable newer APIs including suggested extensions to POSIX for parallelism, and flexible I/O benchmarks that mimic real and dynamic I/O behavior of science and engineering applications. The PIs propose innovative techniques to optimize data accesses that utilize the understanding of high-level access patterns, and use that information through middleware and file systems to enable optimizations. Specifically, the objectives are to (1) design and develop middleware I/O optimizations and cache system that are able to capture small, unaligned, irregular I/O accesses from large number of processors and uses access pattern information to optimize for I/O; (2) incorporate these optimizations in MPICH2's MPI-IO implementation to make them available to a large number of users; (3) design scalable parallel file system techniques and optimizations including a versioning parallel file system, programmable and adaptable consistency semantics, layout optimizations, and self-tuning capabilities; (4) design and evaluate enhanced APIs for file system scalability, particularly for recently proposed enhancements to the POSIX interface (API) to enable highperformance parallel I/O; and (5) develop flexible, execution oriented and scalable I/O benchmarks that mimic the I/O behavior of real science, engineering and bioinformatics applications.

对于数据密集型应用程序，I/O和存储层是非常关键的，并且经常被忽视，但它们不仅成为获得可扩展性能的瓶颈，而且还成为系统和应用科学家利用率和生产力的瓶颈。沿着计算能力，I/O的可扩展软件，以及所需容量和性能的存储必须被开发，以解决应用程序的数据密集性质，并在高端系统的性能和生产力方面获得好处。特别是，这个建议的主要目标是设计和实现新的I/O中间件技术和优化，并行文件系统技术，可扩展到超大规模的系统，设计和开发的技术，有效地使新的API，包括建议的扩展POSIX的并行性，灵活的I/O基准，模仿科学和工程应用程序的真实的和动态的I/O行为。PI提出了利用对高级访问模式的理解来优化数据访问的创新技术，并通过中间件和文件系统使用该信息来实现优化。具体地说，目标是：（1）设计和开发中间件I/O优化和缓存系统，能够捕获来自大量处理器的小的、未对齐的、不规则的I/O访问，并使用访问模式信息来优化I/O;（2）将这些优化结合到MPICH 2的MPI-IO实现中，使它们对大量用户可用;（3）设计可伸缩并行文件系统技术和优化，包括版本化并行文件系统、可编程和自适应一致性语义、布局优化和自调优能力;（4）设计和评估增强的API，以实现文件系统的可扩展性，特别是最近提出的对POSIX接口（API）的增强，以实现高性能并行I/O;以及（5）开发灵活的、面向执行的和可扩展的I/O基准，其模仿真实的科学、工程和生物信息学应用的I/O行为。