Efficient Distributed Algorithms for Convolutional Neural Networks
Efficient Distributed Algorithms for Convolutional Neural Networks
复制标题
DOI：
10.1145/3409964.3461828
复制
发表时间：
2021-05
期刊：
影响因子：
--
通讯作者：
Rui Li;Yufan Xu;Aravind Sukumaran-Rajam;A. Rountev;P. Sadayappan
中科院分区：
文献类型：
--
作者：
Rui Li;Yufan Xu;Aravind Sukumaran-Rajam;A. Rountev;P. Sadayappan
已经开发了几种有效的分布式算法用于矩阵-矩阵乘法：3D算法，2D SUMMA算法和2.5D算法。这些算法中的每一个都是独立构思的，它们权衡了每个节点所需的存储器和节点间的数据通信量。卷积神经网络（CNN）计算可以被视为与邻域模板计算相结合的矩阵乘法的推广。我们为CNN开发了通信高效的分布式存储器算法，类似于矩阵-矩阵乘法的2D/2.5D/3D算法。
Several efficient distributed algorithms have been developed for matrix-matrix multiplication: the 3D algorithm, the 2D SUMMA algorithm, and the 2.5D algorithm. Each of these algorithms was independently conceived and they trade-off memory needed per node and the inter-node data communication volume. The convolutional neural network (CNN) computation may be viewed as a generalization of matrix-multiplication combined with neighborhood stencil computations. We develop communication-efficient distributed-memory algorithms for CNNs that are analogous to the 2D/2.5D/3D algorithms for matrix-matrix multiplication.