SHF: Small: Expediting the Execution of Machine Learning Applications on Multi-GPU Infrastructure with Architecture Awareness and Runtime Support

深度神经网络（dnn）已经成为解决对象分类、自动驾驶汽车、自然语言处理等现实世界问题的最流行的机器学习技术之一。由于问题规模和复杂性的不断增长，深度神经网络模型的训练和推理越来越耗时，需要大量的计算资源。因此，多gpu基础架构是一个理想的平台，已广泛应用于现代深度神经网络任务。然而，交付的DNN执行可伸缩性由于架构上的不了解和缺乏易于使用的运行时支持而受到严重限制。本研究揭示并解决了深度神经网络执行的架构瓶颈。这项研究的结果有望在多gpu基础设施上实现可扩展的深度神经网络执行。该项目的教育和推广部分包括(i)将多gpu基础设施整合到研究生水平的计算机体系结构课程中的新课程项目；（ii）通过高级顶点项目课程和PI研究所的外展计划，让本科生参与研究活动；（三）提高女性和少数族裔学生在计算机体系结构、计算机科学和工程领域的参与度和可见度。这项研究旨在揭示和解决在多gpu上执行DNN的架构瓶颈。具体来说：1)它将地址转换确定为多gpu性能的重要瓶颈。它为在多gpu基础设施上执行单租户和多租户DNN重新设计了翻译Lookaside Buffer （TLB）层次结构和页表遍历。2)研究了现代深度神经网络应用中数据并行性和模型并行性的数据移动开销。它提出了架构感知的数据蒸馏和基于神经元的模型划分来减少数据移动开销。3)提出了一个运行时框架，通过增强可编程性来促进多gpu的使用，允许在执行过程中动态自动地生成虚拟内核到物理内核。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep Neural Networks (DNNs) have become one of the most popular machine-learning techniques for solving real-world problems in object classification, autonomous vehicles, natural language processing, etc. Due to the ever-growing problem size and complexity, the training and inference of DNN models are increasingly time-consuming and require enormous computing resources. As such, multi-GPU infrastructure is a desirable platform that has been widely used in modern DNN tasks. However, the delivered DNN execution scalability is severely limited due to architectural unawareness and lacking easy-to-use runtime support. This research uncovers and addresses the architectural bottlenecks of DNN executions. The outcome of this research is expected to achieve scalable DNN executions on multi-GPU infrastructure. The educational and outreach components of this project include (i) new course projects on multi-GPU infrastructure integrated into graduate-level computer architecture courses; (ii) engaging undergraduate students in the research activities through senior Capstone project courses and an outreach program at PI’s institute; and (iii) increasing the participation and visibility of female and minority students in computer architecture, computer science, and engineering.This research is set to uncover and address the architectural bottlenecks of DNN executions on multi-GPUs. Specifically: 1) It identifies address translation as an essential bottleneck in multi-GPU performance. It redesigns the Translation Lookaside Buffer (TLB) hierarchy and the page table walk for both single-tenant and multi-tenant DNN executions on multi-GPU infrastructure. 2) It investigates the data-movement overheads in data parallelism and model parallelism of modern DNN applications. It proposes architecture-aware data distillation and neuron-based model partitioning to mitigate the data movement overheads. 3) It proposes a runtime framework that fosters the usage of multi-GPUs through enhanced programmability, which allows dynamic and automatic virtual kernel to physical kernel generation during execution.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.