Channel and filter parallelism for large-scale CNN training
Channel and filter parallelism for large-scale CNN training
复制标题
DOI：
10.1145/3295500.3356207
复制
发表时间：
2019-11
期刊：
影响因子：
--
通讯作者：
Nikoli Dryden;N. Maruyama;Tim Moon;Tom Benson;M. Snir;B. V. Essen
中科院分区：
文献类型：
--
作者：
Nikoli Dryden;N. Maruyama;Tim Moon;Tom Benson;M. Snir;B. V. Essen
随着数据集越来越大，模型越来越复杂，需要加速大规模CNN训练，以保持训练时间合理。现有框架主要使用数据并行性进行扩展，但这受到mini-batch大小的限制，不能任意增长。我们介绍了三种划分通道或过滤数据的算法，以利用超出样本维度的并行性。此外，他们对卷积层的参数进行了划分，用分段的全约简取代了全局的全约简——在不相交的处理器集之间更小、并发的全约简。这些算法具有很强的可伸缩性，减少了通信开销，减少了内存压力，使训练非常广泛的cnn成为可能。我们展示了改进的强缩放和弱缩放，包括残余网络的训练时间减少了4.1倍，allreduce开销减少了4倍。我们还表明，更宽的模型在ImageNet上提供了更高的准确性。我们研究了当前算法的局限性，并为未来大规模深度学习框架的优化提供了方向。
Accelerating large-scale CNN training is needed to keep training times reasonable as datasets grow larger and models become more complex. Existing frameworks primarily scale using data-parallelism, but this is limited by the mini-batch size, which cannot grow arbitrarily. We introduce three algorithms that partition channel or filter data to exploit parallelism beyond the sample dimension. Further, they partition the parameters of convolutional layers, replacing global all reduces with segmented allreduces---smaller, concurrent allreduces among disjoint processor sets. These algorithms enable strong scaling, reduced communication overhead, and reduced memory pressure, enabling training of very wide CNNs. We demonstrate improved strong and weak scaling, including up to 4.1x reductions in training time for residual networks and 4x reductions in allreduce overhead. We also show that wider models provide improved accuracy on ImageNet. We study the current limitations of our algorithms and provide a direction for future optimizations of large-scale deep learning frameworks.