Scale-Out Acceleration for Machine Learning
Scale-Out Acceleration for Machine Learning
复制标题
DOI：
10.1145/3123939.3123979
复制
发表时间：
2017-10
期刊：
影响因子：
--
通讯作者：
Jongse Park;Hardik Sharma;Divya Mahajan;J. Kim;Preston Olds;H. Esmaeilzadeh
中科院分区：
文献类型：
--
作者：
Jongse Park;Hardik Sharma;Divya Mahajan;J. Kim;Preston Olds;H. Esmaeilzadeh
机器学习（ML）算法的规模和复杂性的不断增长导致了分布式通用系统的普遍使用。在一个相当分散的努力中，社区主要关注用于学习的高性能单节点加速器。这项工作桥接了这两种范式，并提供了CoSMIC，这是一个完整的计算堆栈，包括语言，编译器，系统软件，模板架构和电路生成器，可以实现大规模学习的可编程加速。CoSMIC使程序员能够利用FPGA和可编程ASIC（P-ASIC）从高级和数学特定领域语言（DSL）中进行横向扩展加速。尽管如此，CoSMIC不需要程序员深入研究系统软件开发或硬件设计的繁重任务。CoSMIC通过集成一种新颖的多线程模板加速器架构和一个从其高级DSL生成硬件和软件代码的内聚堆栈，实现了效率、自动化和可编程性这三个相互冲突的目标。CoSMIC可以加速广泛的学习算法，这些算法通常使用梯度下降的并行变体进行训练。关键是将学习算法的部分梯度计算分布在横向扩展系统的加速器增强节点上。此外，CoSMIC利用算法的并行性在每个节点内提供多线程加速。多线程允许CoSMIC通过在多线程并行性和单线程性能之间取得平衡，有效地利用现代FPGA/P-ASIC上可用的大量资源。CoSMIC利用ML的算法特性提供了一个专门的系统软件，可以优化任务分配、角色分配、线程管理和节点间通信。我们评估了CoSMIC在不同领域的10种不同机器学习应用中的通用性和效率。平均而言，一个带有UltraScale+ FPGA的16节点CoSMIC比一个带有Xeon处理器的16节点Spark系统提供了18.8倍的加速，而程序员只需编写22-55行代码。与最先进的Spark相比，CoSMIC提供了更高的可扩展性;使用CoSMIC从4个节点扩展到16个节点，可获得2.7倍的改进，而Spark提供1.8倍的改进。这些结果证实了CoSMIC的全栈方法朝着实现机器学习的横向扩展加速迈出了有效而重要的一步。
The growing scale and complexity of Machine Learning (ML) algorithms has resulted in prevalent use of distributed general-purpose systems. In a rather disjoint effort, the community is focusing mostly on high performance single-node accelerators for learning. This work bridges these two paradigms and offers CoSMIC, a full computing stack constituting language, compiler, system software, template architecture, and circuit generators, that enable programmable acceleration of learning at scale. CoSMIC enables programmers to exploit scale-out acceleration using FPGAs and Programmable ASICs (P-ASICs) from a high-level and mathematical Domain-Specific Language (DSL). Nonetheless, CoSMIC does not require programmers to delve into the onerous task of system software development or hardware design. CoSMIC achieves three conflicting objectives of efficiency, automation, and programmability, by integrating a novel multi-threaded template accelerator architecture and a cohesive stack that generates the hardware and software code from its high-level DSL. CoSMIC can accelerate a wide range of learning algorithms that are most commonly trained using parallel variants of gradient descent. The key is to distribute partial gradient calculations of the learning algorithms across the accelerator-augmented nodes of the scale-out system. Additionally, CoSMIC leverages the parallelizability of the algorithms to offer multi-threaded acceleration within each node. Multi-threading allows CoSMIC to efficiently exploit the numerous resources that are becoming available on modern FPGAs/P-ASICs by striking a balance between multi-threaded parallelism and single-threaded performance. CoSMIC takes advantage of algorithmic properties of ML to offer a specialized system software that optimizes task allocation, role-assignment, thread management, and internode communication. We evaluate the versatility and efficiency of CoSMIC for 10 different machine learning applications from various domains. On average, a 16-node CoSMIC with UltraScale+FPGAs offers 18.8× speedup over a 16-node Spark system with Xeon processors while the programmer only writes 22–55 lines of code. CoSMIC offers higher scalability compared to the state-of-the-art Spark; scaling from 4 to 16 nodes with CoSMIC yields 2.7× improvements whereas Spark offers 1.8×. These results confirm that the full-stack approach of CoSMIC takes an effective and vital step towards enabling scale-out acceleration for machine learning.