Asynchronous SGD for DNN training on Shared-memory Parallel Architectures
Asynchronous SGD for DNN training on Shared-memory Parallel Architectures
复制标题
DOI：
10.1109/ipdpsw50202.2020.00168
复制
发表时间：
2020-05
期刊：
影响因子：
--
通讯作者：
Florent Lopez;Edmond Chow;S. Tomov;J. Dongarra
中科院分区：
文献类型：
--
作者：
Florent Lopez;Edmond Chow;S. Tomov;J. Dongarra
提出了一种并行异步随机梯度下降算法。与以前的异步算法不同，我们考虑了梯度更新不是特别稀疏的情况。在MagmaDNN框架的背景下，我们比较了异步实现和传统同步实现的并行效率。对在多核cpu和GPU设备上训练深度神经网络进行了测试。
We present a parallel asynchronous Stochastic Gradient Descent algorithm for shared memory architectures. Different from previous asynchronous algorithms, we consider the case where the gradient updates are not particularly sparse. In the context of the MagmaDNN framework, we compare the parallel efficiency of the asynchronous implementation with that of the traditional synchronous implementation. Tests are performed for training deep neural networks on multicore CPUs and GPU devices.