Software-Hardware Co-design of Heterogeneous SmartNIC System for Recommendation Models Inference and Training
Software-Hardware Co-design of Heterogeneous SmartNIC System for Recommendation Models Inference and Training
复制标题
DOI：
10.1145/3577193.3593724
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Anqi Guo;Y. Hao;Chunshu Wu;Pouya Haghi;Zhenyu Pan;Min Si;Dingwen Tao;Ang Li;Martin C. Herbordt;Tong Geng
中科院分区：
文献类型：
--
作者：
Anqi Guo;Y. Hao;Chunshu Wu;Pouya Haghi;Zhenyu Pan;Min Si;Dingwen Tao;Ang Li;Martin C. Herbordt;Tong Geng
深度学习推荐模型(DLRM)在各个领域都有重要的应用，已经发展成为最大、最重要的机器学习应用之一。随着它们的万亿参数必然超过GPU的高带宽内存(HBM)能力，越来越庞大的DLRM需要大规模的多节点系统来进行分布式训练和推理。然而，所有这些都受到所有到所有通信瓶颈的影响，这限制了可扩展性。SmartNIC将计算和通信功能结合在一起，以提供强大的面向网络的异类设备，从而减少通信开销。然而，还没有一种分布式系统设计来充分利用SmartNIC资源来解决DLRM的可扩展性问题。提出了一种软硬件协同设计的异构型SmartNIC系统，克服了分布式DLRMS的通信瓶颈，缓解了内存带宽压力，提高了计算效率。我们提供了一套SmartNIC缓存系统(包括本地缓存和远程缓存)和SmartNIC计算内核的设计，减少了数据移动，减轻了内存查找强度，提高了GPU的计算效率。此外，我们还提出了一种图算法，该算法提高了批量查询的数据局部性，并以更高的数据重用率优化了系统的整体性能。我们的测试表明，该系统的推理延迟加速比为2.1倍，训练吞吐量加速比为1.6倍。
Deep Learning Recommendation Models (DLRMs) are important applications in various domains and have evolved into one of the largest and most important machine learning applications. With their trillions of parameters necessarily exceeding the high bandwidth memory (HBM) capacity of GPUs, ever more massive DLRMs require large-scale multi-node systems for distributed training and inference. However, these all suffer from the all-to-all communication bottleneck, which limits scalability. SmartNICs couple computation and communication capabilities to provide powerful network-facing heterogeneous devices that reduce communication overhead. There has not, however, been a distributed system design that fully leverages SmartNIC resources to address scalability of DLRMs. We propose a software-hardware co-design of a heterogeneous SmartNIC system that overcomes the communication bottleneck of distributed DLRMs, mitigates the pressure on memory bandwidth, and improves computation efficiency. We provide a set of SmartNIC designs of cache systems (including local cache and remote cache) and SmartNIC computation kernels that reduce data movement, relieve memory lookup intensity, and improve the GPU's computation efficiency. In addition, we propose a graph algorithm that improves the data locality of queries within batches and optimizes the overall system performance with higher data reuse. Our evaluation shows that the system achieves 2.1× latency speedup for inference and 1.6× throughput speedup for training.