EAGER: From Theory to Practice of Elastic Interval Runtime Schedulers

遍布地球仪的数据中心和高性能计算系统（HPC）构成了现代计算服务和应用的支柱，例如互联网搜索、科学发现、电子邮件、天气预测、社交媒体平台和数据共享。数据中心和HPC内的资源（计算、内存和网络）编排一直是一个挑战，最主要的原因是无法100%准确地预测应用程序需求和资源利用率。因此，资源编排方法对应用需求（例如，作业到达率和作业完成时间）进行关键假设以利用资源。这种假设往往导致瓶颈或利用不足。该项目研究一类新的资源编排方法，称为弹性间隔调度器（EIRIS），不承担对作业到达率或完成时间或应用程序并行性界限的假设。该项目将在坚实的数学基础上为EIRIS开发实用的解决方案。预计该项目将对数据中心和HPC的调度理论和运行时系统产生深远的影响。直接受益于这项工作的应用包括（但不限于）：云游戏，视频编码和视频转码。该项目的研究成果将被纳入计算机系统的研究生班。此外，每月播客，题为计算机系统的概念，将广播给密西西比高中学生的目标，鼓励他们追求在计算机和信息科学和工程教育。该项目将开发的数学基础，理论界，和可翻译的运行时间为EIRIS。该项目将分为三个交叉的重点：（1）EIRIS的数学基础，利用群论的概念和运算：（a）开发分析的预言，（B）推导理论下限，（c）尝试进一步完善下限;（2）通过利用装箱算法的概念，为EIRIS开发一套资源编排方法，以开发主要依赖于以下内容的程序：（a）实际作业持续时间和（B）剩余作业持续时间;以及（3）通过实现（a）分布式资源管理系统、（B）元数据服务器和（c）用于可再现工件的包，用于G-Hadoop中的EIRIS的自主运行时。所有的推进器将在大规模计算机系统上进行部署和测试，以供演示。该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估的支持。

Data centers and high-performance computing systems (HPC) across the globe form the backbone of modern-day computing services and applications, such as Internet searches, scientific discoveries, emails, weather predictions, social media platforms, and data sharing. Resource (computational, memory, and network) orchestration within data centers and HPC, is, and always has been, a challenge, the foremost reason being that application requirements and resource utilization cannot be predicted with 100% accuracy. Consequently, resource-orchestration methodologies undertake critical assumption on the application needs (for example, job arrival rates and job completions times) to utilize resources. More than often, such assumptions lead to bottlenecks or underutilization. This project investigates a new class of resource-orchestration methodologies, called the Elastic Interval Runtime Schedulers (EIRIS), that do not undertake assumptions on the job arrival rates or completion times or bounds on the application parallelism. The project will develop practical solutions for EIRIS anchored in solid mathematical underpinnings. The project is expected to have a profound impact on the scheduling theory and runtime systems for data centers and HPC. Applications immediately benefiting from this work include (but are not limited to): cloud gaming, video coding, and video transcoding. The project research findings will be integrated into a graduate class on computer systems. Moreover, a monthly podcast, entitled concepts in computer systems, will be broadcasted to Mississippi high school students with the goal to encourage them to pursue education in computer and information science and engineering.The project will develop the mathematical underpinnings, theoretical bounds, and translatable runtime for EIRIS. The project will be divided into three interleaving thrusts focusing on: (1) mathematical underpinnings for EIRIS by utilizing concepts and operations from group theory to: (a) develop an oracle for analyses, (b) derive theoretical lower bounds, and (c) make an attempt to further refine the lower bounds; (2) developing a set of resource-orchestration methodologies for EIRIS by utilizing concepts from bin packing algorithms to develop procedures predominantly relying on: (a) actual job duration times and (b) remaining job duration times; and (3) an autonomic runtime for EIRIS in G-Hadoop by implementing (a) distributed resource management systems, (b) metadata servers, and (c) packages for reproducible artifacts. All thrusts will be deployed and tested on large-scale computer systems for demonstration.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.