CRAM: A Congestion-Aware Resource and Allocation Manager for Data-Intensive High-Performance Computing

该项目将开发一个用于数据密集型高性能计算(HPC)的作业调度和资源分配系统，该系统基于系统的异类资源的拥塞定价。这将资源管理的概念扩展到处理之外：它在作业之间分配内存、磁盘I/O和网络。这项研究将克服以处理器为中心的资源管理的严重缺陷，这种资源管理将大量的集群和超级计算机资源浪费在数据密集型工作负载上，例如，I/O带宽控制着许多现代高性能计算应用的性能，但目前既没有分配也没有管理。这项研究将开发以下技术：(1)调整高性能计算作业的并行度，以避免拥塞和浪费；(2)支持优先级较低的分配弹性作业，这些作业可以在任意数量的节点上调度以消耗未分配的资源碎片；(3)共同调度批处理工作负载，这些工作负载使用由于前台作业的不对称利用和时间转移而未占用的系统资源。这些技术将作为开放源码资源管理框架的扩展被实施和支持供公众免费使用。如果广泛使用，该软件有可能更好地利用国家对高性能计算设施的投资。

This project will develop a job scheduling and resource allocation system for data-intensive high-performance computing (HPC) based on the congestion pricing of a systems' heterogeneous resources. This extends the concept of resource management beyond processing: it allocates memory, disk I/O, and the network among jobs. The research will overcome the critical shortcomings of processor-centric resource management, which wastes huge portions of cluster and supercomputer resources for data-intensive workloads, e.g. I/O bandwidth governs the performance of many modern HPC applications but, at present, it is neither allocated nor managed. The research will develop techniques that (1) reconfigure the degree of parallelism of HPC jobs to avoid congestion and wastage, (2) support lower-priority, allocation elastic jobs that can be scheduled on arbitrary numbers of nodes to consume unallocated resource fragments, and (3) co-schedule batch-processing workloads that use system resources that are unoccupied due to asymmetric utilization and temporal shifts in the foreground jobs. These techniques will be implemented and supported for free public use as extensions to an open-source resource-management framework. If used broadly, the software has the potential to provide much better utilization of the national investment in HPC facilities.