System Software Support for Input-Output on Parallel Computing

信息处理以及科学和工程中的大量应用都需要处理大量的数据。特别是在科学和工程领域，所谓的“重大挑战应用”就属于这一类。因此，很明显，I/O性能远高于CPU或通信性能可能是未来并行计算系统的限制因素。许多软件和硬件组件在I/O操作中起着关键作用，其中任何一个都可能导致严重的瓶颈。磁盘系统的性能和CPU(和互连)性能之间的差异需要使用大规模并行(和支持软件)来解决I/O子系统中的可扩展性和性能问题。但I/O系统中的并行性给系统软件带来了复杂性，包括编译器、运行时系统和文件系统。这项研究解决了并行I/O领域中的许多系统软件问题。为了改善I/O瓶颈，以下是必需的：1)用于复杂运行时系统的技术，其结合了集合I/O、数据重用和预取的概念；2)编译器分析，以识别各种优化，包括数据访问重组、重叠计算和I/O的机会、以及数据分级，以处理用于核外计算的有限存储器；以及3)文件系统优化，以识别用于协调作为并行程序一部分的进程的访问的集合访问模式，并将这些访问模式用于预取、缓冲区分配和替换策略。传统的文件系统接口缺乏实现上述优化所需的语义。因此，有必要开发编译器和运行时系统之间的接口，以及运行时系统和文件系统之间的接口，以促进关于访问模式的语义信息流，以实现优化。这项研究致力于设计和开发编译器分析技术，运行时系统的机制和算法，以及文件系统的接口和优化，以实现并行计算机中的高性能I/O。这些技术正在通过在并行系统上实施来进行评估。***

A large number of applications from information processing as well as science and engineering require processing large amounts of data. Particularly, in the science and engineering domain, the so called ``grand challenge applications'' fall in this category. Thus, it has become apparent that I/O performance much more than CPU or communication performance may be the limiting factor in future parallel computing systems. Many software and hardware components play a key role in I/O operations and any one of them can cause severe bottlenecks. The disparity between the performance of disk systems and CPU (and interconnect) performance requires the use of large-scale parallelism (and supporting software) to address the scalability and performance problems in I/O subsystems. But parallelism in the I/O systems introduces complexities for the system software including compilers, runtime systems and file systems. This research addresses many system software problems in the parallel I/O domain. To ameliorate the I/O bottlenecks, the following are necessary: 1) techniques for sophisticated runtime systems that incorporate the notion of collective I/O, data reuse and prefetching; 2) compiler analysis to identify various optimizations including data access reorganization, opportunities for overlapping computations and I/O, and data staging to handle limited memory for out-of-core computations; and 3) file systems optimizations to recognize collective access patterns for coordinating accesses of processes that are part of parallel programs, and using these access patterns for prefetching, buffer allocation and replacement strategies. Traditional interfaces for file systems lack the necessary semantics to permit the above optimizations. Thus, it is necessary to develop interfaces between the compiler and the runtime systems, and the runtime system and the file systems, that facilitate the flow of semantic information about access patterns to enable optimizations. This res earch pursues the design and development of techniques for compiler analysis, mechanisms and algorithms for runtime systems, and interfaces and optimizations for file systems to enable high- performance I/O in parallel computers. These techniques are being evaluated by implementing them on parallel systems. ***