SHF: CSR: Small: Toward Smart HPC through Active Learning and Intelligent Scheduling

随着高性能计算(HPC)规模的持续增长，除了对性能的追求外，能源和弹性成为首要关注的问题。这些问题要求在系统堆栈的许多方面进行重大更改，包括资源管理和作业调度。为了利用Extreme Scale系统的巨大潜力，该项目旨在将智能融入资源管理和作业调度。更具体地说，它将开发一个名为SPAR(Scheduling for Performance，Energy，and Resilience Efficiency)的框架，用于动态优化三维性能、能量和弹性调度。研究集中在两个方面：一是主动学习，自动从应用程序和系统数据中提取有价值的性能、能量和弹性模式和权衡；二是智能调度，在资源管理和调度中改进和控制性能、弹性和能效。正在开发一个事件驱动的调度模拟器，用于全面评估调度策略及其聚合效果。模拟器以及系统日志将在开放源码许可下提供给广大社区。该项目创造了提高系统生产率的关键技术，并取得了对智能HPC至关重要的重要进展。此外，该项目开发的学习技术对其他涉及国家利益的大数据问题也很有用。该教育计划加强了本科生和研究生课程，扩大了代表人数不足的群体的参与。

As high performance computing (HPC) continues to grow in scale, energy and resilience become first-class concerns, in addition to the pursuit of performance. These concerns demand significant changes in many aspects of the system stack including resource management and job scheduling. In order to harness the great potential of extreme scale systems, this project aims to incorporate intelligence into resource management and job scheduling. More specifically, it will develop a framework named SPEaR (Scheduling for Performance, Energy, and Resilience efficiency) for dynamically optimizing the three-dimensional performance, energy, and resilience scheduling. The research focuses on two thrusts: one is active learning to automatically extract valuable performance, energy, and resilience patterns and tradeoffs out of application and system data, and the other is intelligent scheduling to improve and control performance, resilience, and energy efficiency in resource management and scheduling. An event-driven scheduling simulator is being developed for comprehensively evaluating scheduling policies and their aggregate effects. The simulator, along with system logs, will be made available to the broad community under an open source license. This project creates critical technologies to promote system productivity and makes important advances essential toward smart HPC. Additionally, the learning techniques developed in this project are useful to other big data problems of national interests. The education plan enhances the undergraduate and graduate curricula and broadens the participation from underrepresented groups.