Collaborative Research: OAC Core: Improving Utilization of High-Performance Computing Systems via Intelligent Co-scheduling

该项目旨在通过在系统中的同一组节点上调度多个作业来提高高性能计算系统的效率，通常称为协同调度。这与当前的做法不同，在当前的做法中，节点一次专用于一个作业，这导致可预测的执行时间，但系统资源的使用效率低下。为了使这一点切实可行，该项目将开展分析，以确定如何进行联合调度，从而提高整个系统的效率，同时尽量减少对个别应用程序的性能影响。特别是，我们的目标是共同调度的工作，可以共存，而不会争夺类似的资源节点上。 该项目所做的工作将有助于提高高性能系统的效率，这将影响气候/天气、可再生能源和国家安全等应用领域。这项工作将在劳伦斯利弗莫尔和桑迪亚国家实验室的系统上实施和验证，然后过渡到这些国家实验室使用的软件中。该项目还将通过将本研究中的技术整合到PI机构的并行和分布式计算课程中来对教育产生影响。此外，该项目将在两个为西班牙裔学生服务的机构进行，而PI有为代表性不足的学生提供建议的历史;该项目将通过招募西班牙裔本科生参与该项目并将他们送到国家实验室实习来扩大对计算的参与。在高端计算设施中，长期存在的抽象是一个提交的作业被分配给一组专用节点。然而，这使得系统的效率低得多，因为每个节点的资源更多，往往会被低效地使用。此外，对高端系统的需求正在增加，将节点专用于作业会增加作业周转时间并降低整体系统吞吐量。 解决这个问题的一种方法是让超级计算机中心打破目前的惯例，即为每个作业分配超级计算机的一个私有的、孤立的部分。 该项目的智力价值是三方面的。首先，将开发新的配置文件分析，这将揭示由于共享节点对工作的影响。其次，将开发新的统计投影技术，预测正在利用共享节点的作业的缩放行为。第三，将设计新的作业级调度技术，使用干扰分析和预测来选择一组共享节点，这将导致良好的作业周转时间和最大化系统吞吐量。该项目的广泛影响是多方面的。 该项目将有助于提高高性能系统的效率，这将有利于广泛的应用，包括气候/天气预测，核能和国家安全。 通过与Lawrence Livermore和Sandia国家实验室的长期合作，PI将在LLNL和SNL系统上实施和验证技术，并将这些技术过渡到国家实验室未来的资源管理器中。此外，这两个PI将通过招募西班牙裔本科生参与该项目并将他们送到国家实验室实习来扩大对计算的参与。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

This project is aimed at increasing efficiency of high-performance computing systems by scheduling multiple jobs on the same set of nodes in a system, generally called co-scheduling. This is a break from current practice in which nodes are dedicated to one job at a time, which results in predictable execution time but inefficient use of system resources. To make this practical, the project will develop analyses to determine how to carry out co-scheduling such that overall system efficiency is improved while the performance impact on individual applications is minimized. In particular, the goal is to co-schedule jobs that can co-exist without contending for similar resources on the nodes. The work done in this project will help achieve better efficiency on high-performance systems, which will impact application domains such as climate/weather, renewable energy, and national security. The work will be implemented and validated on systems at Lawrence Livermore and Sandia National Laboratories and then transitioned into software that will be used at these national laboratories. The project will also have an impact on education by integrating the techniques in this research into courses covering parallel and distributed computing at the PIs' institutions. In addition, the project will take place at two Hispanic-serving institutions, and the PIs have a history of advising under-represented students; the project will broaden participation in computing by recruiting Hispanic undergraduates to work on the project and sending them to national laboratories for internships.The long-standing abstraction at high-end computing facilities is one of a submitted job being allocated a set of dedicated nodes. However, this makes systems much less efficient, as there are more per-node resources that will often be used inefficiently. In addition, the demand for high-end systems is increasing and dedicating nodes to jobs can increase job turnaround time and decrease overall system throughput. One way to address this problem is for supercomputer centers to break from the current common practice of assigning each job a private, isolated portion of a supercomputer. The intellectual merit of the project is three-fold. First, novel profile analyses will be developed that will reveal the effects on jobs due to sharing nodes. Second, novel statistical projection techniques will be developed that predict scaling behavior of jobs that are utilizing shared nodes. Third, new job-level scheduling techniques will be designed that use the interference analysis and projections to choose a set of shared nodes that will lead to good job turnaround time and maximize system throughput. The broader impact of the project is multifold. This project will help achieve better efficiency on high-performance systems, which will benefit a broad range of applications that includes climate/weather prediction, nuclear energy, and national security. Through a long-standing collaboration with both Lawrence Livermore and Sandia National Laboratories, the PIs will implement and validate the techniques on LLNL and SNL systems as well as transition the techniques into future resource managers at the national laboratories. In addition, both PIs will broaden participation in computing by recruiting Hispanic undergraduates to work on the project and sending them to national labs for internships.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.