SHF: CSR: Small: Toward Smart HPC through Active Learning and Intelligent Scheduling

随着高性能计算（HPC）在规模上的持续增长，除了追求性能之外，能源和弹性成为首要关注的问题。这些问题要求在系统堆栈的许多方面进行重大更改，包括资源管理和作业调度。为了利用极端规模系统的巨大潜力，该项目旨在将智能整合到资源管理和作业调度中。更具体地说，它将开发一个名为SPEaR（调度性能、能源和弹性效率）的框架，用于动态优化三维性能、能源和弹性调度。研究重点集中在两个方面：一是主动学习，从应用和系统数据中自动提取有价值的性能、能源和弹性模式和权衡；二是智能调度，以提高和控制资源管理和调度中的性能、弹性和能源效率。为了全面评估调度策略及其总体效果，正在开发一个事件驱动的调度模拟器。模拟器和系统日志将在开源许可下提供给广泛的社区。该项目创造了提高系统生产力的关键技术，并对智能高性能计算取得了重要进展。此外，本项目开发的学习技术对其他涉及国家利益的大数据问题也有借鉴意义。该教育计划加强了本科和研究生课程，扩大了代表性不足群体的参与。

As high performance computing (HPC) continues to grow in scale, energy and resilience become first-class concerns, in addition to the pursuit of performance. These concerns demand significant changes in many aspects of the system stack including resource management and job scheduling. In order to harness the great potential of extreme scale systems, this project aims to incorporate intelligence into resource management and job scheduling. More specifically, it will develop a framework named SPEaR (Scheduling for Performance, Energy, and Resilience efficiency) for dynamically optimizing the three-dimensional performance, energy, and resilience scheduling. The research focuses on two thrusts: one is active learning to automatically extract valuable performance, energy, and resilience patterns and tradeoffs out of application and system data, and the other is intelligent scheduling to improve and control performance, resilience, and energy efficiency in resource management and scheduling. An event-driven scheduling simulator is being developed for comprehensively evaluating scheduling policies and their aggregate effects. The simulator, along with system logs, will be made available to the broad community under an open source license. This project creates critical technologies to promote system productivity and makes important advances essential toward smart HPC. Additionally, the learning techniques developed in this project are useful to other big data problems of national interests. The education plan enhances the undergraduate and graduate curricula and broadens the participation from underrepresented groups.