CRAM: A Congestion-Aware Resource and Allocation Manager for Data-Intensive High-Performance Computing

本项目将基于系统异构资源的拥塞定价，为数据密集型高性能计算（HPC）开发一个作业调度和资源分配系统。这将资源管理的概念扩展到处理之外：它在作业之间分配内存、磁盘I/O和网络。该研究将克服以处理器为中心的资源管理的关键缺点，这些缺点浪费了大量的集群和超级计算机资源用于数据密集型工作负载，例如I/O带宽控制着许多现代HPC应用程序的性能，但目前它既没有分配也没有管理。该研究将开发以下技术：(1)侦察技术；确保HPC作业的并行度以避免拥塞和浪费，(2)支持低优先级的分配弹性作业，这些作业可以调度到任意数量的节点上，以消耗未分配的资源片段，(3)协同调度批处理工作负载，这些工作负载使用由于不对称利用和前台作业的时间转移而未占用的系统资源。这些技术将作为开源资源管理框架的扩展实现和支持，供公众免费使用。如果广泛使用，该软件有可能更好地利用国家在高性能计算设施上的投资。

This project will develop a job scheduling and resource allocation system for data-intensive high-performance computing (HPC) based on the congestion pricing of a systems' heterogeneous resources. This extends the concept of resource management beyond processing: it allocates memory, disk I/O, and the network among jobs. The research will overcome the critical shortcomings of processor-centric resource management, which wastes huge portions of cluster and supercomputer resources for data-intensive workloads, e.g. I/O bandwidth governs the performance of many modern HPC applications but, at present, it is neither allocated nor managed. The research will develop techniques that (1) reconfigure the degree of parallelism of HPC jobs to avoid congestion and wastage, (2) support lower-priority, allocation elastic jobs that can be scheduled on arbitrary numbers of nodes to consume unallocated resource fragments, and (3) co-schedule batch-processing workloads that use system resources that are unoccupied due to asymmetric utilization and temporal shifts in the foreground jobs. These techniques will be implemented and supported for free public use as extensions to an open-source resource-management framework. If used broadly, the software has the potential to provide much better utilization of the national investment in HPC facilities.