A Decoupled I/O Architecture for Data-Intensive Cluster Computing

数据密集型集群计算的解耦 I/O 架构

• 批准号: 0083497
• 负责人: Tzi-Cker Chiueh
• 金额: $ 12.8万
• 依托单位: SUNY at Stony Brook
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2001
• 资助国家: 美国
• 起止时间: 2001-08-15 至 2004-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0083497&HistoricalAwards=false
• 关键词: Decoupled; Architecture; Data; Intensive; Cluster

Cluster computing exploits the exponential growth in processing power, memory/disk capacity, and network link speed of commodity hardware developed in the PC industry to achieve excellent price-performance, and is becoming a major platform for scientific computation, Web page and file servicing, as well as large-scale information analysis applications such as data mining and knowledge discovery. A distinct characteristicshared by many emerging workloads on PC clusters is the requirement for intensive disk data accesses during program computation. The fundamental technique to address the performance problems associated with disk I/O is to overlap disk access with program computation so that the I/O delays are completely masked. Disk prefetching is one incarnation of this technique. In this project, we propose an automatic application-specific file prefetching scheme that exploits specific applications' access patterns to the fullest extent and achieves close to perfect prefetching. The key idea is to apply the concept of decoupled architecture, which was originally proposed to bridge the gap between CPU and memory, to overlap computation with disk I/O. Given an application program, the compiler automatically translates it into two threads, one correspondingto the original program (computation thread) and the other including all statements in the original program that are related to disk I/O (prefetch thread). At run time, the prefetch thread is scheduled to be suficiently ahead of the computation thread so that all the file access requests of the computation thread are satisfied in the file system cache, which is populated by the prefetch thread anticipatively. To validate the decoupled I/O architecture, we will design, implement, and evaluate an active parallel disk I/O subsystem called Platypus that embodies the application-specific file prefetching scheme and that is designed to be a modular building block for scalable PC cluster systems that aim at data-intensive computing. Platypus consists of a source-to-source translator to generate computation and prefetch threads from a SPMD parallel programautomatically, a run-time thread scheduler that coordinates the execution of prefetch threads on the I/O nodes, and a cache manager that maximizes main memory utilization efficiency by optimally balancing the benefits of file caching and prefetching.

集群计算利用PC工业中开发的商品硬件的处理能力、存储器/磁盘容量和网络链接速度的指数增长来实现优异的性价比，并且正在成为科学计算、网页和文件服务以及大规模信息分析应用（如数据挖掘和知识发现）的主要平台。PC集群上的许多新兴工作负载共享的一个明显特征是在程序计算期间需要密集的磁盘数据访问。解决与磁盘I/O相关的性能问题的基本技术是将磁盘访问与程序计算重叠，以便完全屏蔽I/O延迟。磁盘预取是这种技术的一个具体体现。在这个项目中，我们提出了一个自动的应用程序特定的文件预取计划，利用特定的应用程序的访问模式，以最大限度地实现接近完美的预取。其核心思想是应用解耦体系结构的概念，该概念最初是为了弥合CPU和内存之间的差距而提出的，以重叠计算与磁盘I/O。给定一个应用程序，编译器自动将其转换为两个线程，一个对应于原始程序（计算线程），另一个包括原始程序中与磁盘I/O相关的所有语句（预取线程）。在运行时，预取线程被调度为充分领先于计算线程，以便计算线程的所有文件访问请求在文件系统缓存中得到满足，该文件系统缓存由预取线程预先填充。为了验证解耦的I/O体系结构，我们将设计，实现和评估一个活跃的并行磁盘I/O子系统，称为Platypus，体现了应用程序特定的文件预取计划，并被设计为可扩展的PC集群系统，旨在数据密集型计算的模块化构建块。Platypus包括一个源到源翻译器，用于自动从SPMD并行处理器生成计算和预取线程，一个运行时线程调度器，用于协调I/O节点上预取线程的执行，以及一个缓存管理器，用于通过最佳平衡文件缓存和预取的好处来最大限度地提高主存利用率。