CAREER: Achieving Real-Time Machine Learning with Sparsification-Compilation Co-design

机器学习(ML)，特别是深度学习(DL)近年来取得了巨大的成功，特别是不同类型的深度神经网络(DNN)的使用。各种DNN是许多关键应用的最新基础和核心使能器，如机器人、高质量视频流处理、增强现实、可穿戴设备、智能医疗设备等。要实现高精度，通常需要DNN具有庞大而复杂的模型结构，这也意味着对训练和推理步骤的高计算要求。在现代高性能计算(HPC)节点或群集上加速培训过程，以及在低端节能设备上加快推理过程都已成为主要挑战。本项目关注这一问题，将DNN培训和推理视为需要利用可用的多级并行性、复杂的内存层次结构和设备异构性的HPC工作负载；同时通过编译器自动化优化。如果该项目成功，它将首次实现许多边缘设备的实时机器学习，使基于ML的终端应用程序获得更大成功，这些应用程序对社会、经济和其他科学和工程领域具有重要意义。这个项目还将对教育和提高多样性做出几项贡献，包括：(1)在ML课程中引入HPC，并在本科系统和研究生研究课程中引入ML工作量优化经验，特别是通过有趣的演示视频；(2)向本科生延伸，目标是激发(系统)研究的兴趣，并向K-12延伸，目标是吸引未被充分代表的群体学习计算机科学。该项目解决上述挑战的关键思想是稀疏-汇编联合设计。它首先引入了一种称为细粒度结构剪枝的稀疏化思想，该思想根据一定的细粒度结构剪枝权重，并以更规则的方式保留非零权重。基于这一思想，本项目设计了一种称为层级中间表示(IR)的高级抽象来捕获稀疏信息，目标是实现积极的编译器优化。在成功地将这一想法应用于二维DNN的基础上，该项目开展了一项全面的议程，以充分应用这一方法的好处。首先，它将卷积神经网络和递归神经网络加速与更通用的细粒度结构化剪枝实例和一组增强的基于编译器的自动优化相结合。其次，它通过将编译器优化从推理扩展到剪枝，并利用域属性来执行优化的应用级检查点，改进了剪枝或重新训练过程本身。第三，它扩展了(编译器自动)优化框架，以支持高维和极深的DNN。最后，它探索了在多个DNN在同一设备上共同执行的情况下跨DNN的数据重用。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Machine Learning (ML), particularly Deep Learning (DL), has gained great success in recent years, especially with the use of Deep Neural Networks (DNNs) of different types. Varied DNNs serve as the state-of-the-art foundation and core enabler of many key applications, such as robotics, high-quality video stream processing, augmented reality, wearable devices, smart health devices, etc. Achieving high accuracy typically requires DNNs with large and complex model structures, which also translates into high computing requirements for both training and inference steps. Accelerating the training process on a modern High-Performance Computing (HPC) node or cluster and inference process on a lower-end power-efficient device have both emerged as major challenges. This project focuses on this problem, viewing DNN training and inference as HPC workloads that need to exploit available multi-level parallelism, complex memory hierarchy, and device heterogeneity; while automating the optimizations through a compiler. If this project succeeds, it will, for the first time, enable real-time machine learning for many edge devices, enabling the greater success of ML-based end applications that are important for the society, economy, and other science and engineering areas. This project will also make several contributions towards both education and improving diversity, including: (1) introducing HPC in an ML course, and ML workloads optimization experience in both undergraduate systems and graduate research courses, particularly with interesting demonstration videos; (2) outreaching to undergraduates with the goal of creating interest in (systems) research, and to K-12 with the goal of attracting underrepresented groups to computer science.The key idea of this project to address the above challenge is sparsification-compilation co-design. It first introduces a general sparsification idea called fine-grained structured pruning, which prunes the weights according to certain fine-grained structures and preserves non-zero weights in a more regular way. Based on this idea, this project designs a high-level abstraction called layer-wise intermediate representation (IR) to capture the sparsity information with the goal of enabling aggressive compiler optimizations. Building on a successful application of this idea on two-dimensional DNNs, this project undertakes a comprehensive agenda to fully apply the benefits of this approach. First, it unifies Convolutional Neural Networks and Recurrent Neural Networks acceleration with a more general fine-grained structured pruning instance and a set of enhanced compiler-based automatic optimizations. Second, it improves the pruning or retraining process itself by extending the compiler optimizations from inference to pruning and exploiting domain properties to carry-out optimized application-level checkpointing. Third, it extends the (compiler automated) optimization framework to support high-dimensional and extremely deep DNNs. Finally, it explores data reuse across DNNs for situations where multiple DNNs are co-executed on the same device.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.