CRAM: A Congestion-Aware Resource and Allocation Manager for Data-Intensive High-Performance Computing

本计画将针对资料密集型高效能计算，开发一个以系统异质资源拥挤定价为基础的作业排程与资源分配系统。 这将资源管理的概念扩展到处理之外：它在作业之间分配内存、磁盘I/O和网络。 该研究将克服以处理器为中心的资源管理的关键缺点，该管理将大量集群和超级计算机资源浪费在数据密集型工作负载上，例如I/O带宽控制着许多现代HPC应用程序的性能，但目前，它既没有分配也没有管理。 这项研究将开发技术，（1）侦察fi确定HPC作业的并行度，以避免拥塞和浪费，（2）支持低优先级、分配弹性作业，这些作业可以在任意数量的节点上调度，以消耗未分配的资源片段，以及（3）协同调度批处理工作负载，这些工作负载使用由于前台作业中的不对称利用率和时间偏移而未被占用的系统资源。 这些技术将作为开源资源管理框架的扩展，免费供公众使用。 如果广泛使用，该软件有可能更好地利用国家对HPC设施的投资。

This project will develop a job scheduling and resource allocation system for data-intensive high-performance computing (HPC) based on the congestion pricing of a systems' heterogeneous resources. This extends the concept of resource management beyond processing: it allocates memory, disk I/O, and the network among jobs. The research will overcome the critical shortcomings of processor-centric resource management, which wastes huge portions of cluster and supercomputer resources for data-intensive workloads, e.g. I/O bandwidth governs the performance of many modern HPC applications but, at present, it is neither allocated nor managed. The research will develop techniques that (1) reconfigure the degree of parallelism of HPC jobs to avoid congestion and wastage, (2) support lower-priority, allocation elastic jobs that can be scheduled on arbitrary numbers of nodes to consume unallocated resource fragments, and (3) co-schedule batch-processing workloads that use system resources that are unoccupied due to asymmetric utilization and temporal shifts in the foreground jobs. These techniques will be implemented and supported for free public use as extensions to an open-source resource-management framework. If used broadly, the software has the potential to provide much better utilization of the national investment in HPC facilities.