Automating Dependence-Aware Parallelization of Machine Learning Training on Distributed Shared Memory
Automating Dependence-Aware Parallelization of Machine Learning Training on Distributed Shared Memory
复制标题
DOI：
10.1145/3302424.3303954
复制
发表时间：
2019-03
期刊：
影响因子：
--
通讯作者：
Jinliang Wei;Garth A. Gibson;Phillip B. Gibbons;E. Xing
中科院分区：
文献类型：
--
作者：
Jinliang Wei;Garth A. Gibson;Phillip B. Gibbons;E. Xing
机器学习(ML)训练通常使用数据并行化。数据并行性的一个基本限制是，ML训练期间冲突的(并发)参数访问通常会减少甚至否定额外的并行计算资源所提供的好处。尽管通过仔细调度计算可以避免冲突的参数访问，但现有系统依赖于程序员手动并行化，并且这种并行化何时可能仍然是一个问题。我们介绍了一个在分布式共享内存上自动并行化串行命令式ML程序的系统Orion。Orion的核心是一种静态依赖分析机制，该机制确定依赖保持并行化何时有效，并将循环计算映射到优化的分布式计算调度。我们的评估表明，对于许多ML应用程序，Orion可以在保持关键依赖的同时并行化串行程序，从而实现比数据并行程序快得多的收敛速度，以及与包括模型并行程序在内的最先进的手动并行化程序相匹配的收敛速度和相当的计算吞吐量。
Machine learning (ML) training is commonly parallelized using data parallelism. A fundamental limitation of data parallelism is that conflicting (concurrent) parameter accesses during ML training usually diminishes or even negates the benefits provided by additional parallel compute resources. Although it is possible to avoid conflicting parameter accesses by carefully scheduling the computation, existing systems rely on programmer manual parallelization and it remains a question when such parallelization is possible. We present Orion, a system that automatically parallelizes serial imperative ML programs on distributed shared memory. The core of Orion is a static dependence analysis mechanism that determines when dependence-preserving parallelization is effective and maps a loop computation to an optimized distributed computation schedule. Our evaluation shows that for a number of ML applications, Orion can parallelize a serial program while preserving critical dependences and thus achieve a significantly faster convergence rate than data-parallel programs and a matching convergence rate and comparable computation throughput to state-of-the-art manual parallelizations including model-parallel programs.