A novel memory-efficient deep learning training framework via error-bounded lossy compression
A novel memory-efficient deep learning training framework via error-bounded lossy compression
复制标题
DOI：
10.1145/3437801.3441597
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Sian Jin;Guanpeng Li;S. Song;Dingwen Tao
中科院分区：
文献类型：
--
作者：
Sian Jin;Guanpeng Li;S. Song;Dingwen Tao
由于对预测精度和分析质量的需求不断增长，DNN变得越来越深，越来越宽，越来越非线性。在训练DNN模型时，中间激活数据必须在前向传播期间保存在内存中，然后恢复用于后向传播。传统的内存节省技术，如数据重新计算和迁移，要么遭受高性能开销，或受到特定的互连技术和有限的带宽。在本文中，我们提出了一种新的内存驱动的高性能CNN训练框架，该框架利用误差有界有损压缩来显著降低训练的内存需求，以便训练更大的神经网络。具体来说，我们提供了理论分析，然后提出了一种改进的有损压缩器和自适应方案，动态配置有损压缩误差范围，并调整训练批量大小，以进一步利用节省的内存空间，以实现额外的加速。我们使用四种广泛采用的CNN和ImangeNet数据集来评估我们的设计。结果表明，我们提出的框架可以显着减少训练内存消耗高达13.5倍和1.8倍，分别超过基线训练和最先进的压缩框架，几乎没有或没有准确性损失。全文可在https://arxiv.org/abs/2011.09017上查阅。
DNNs are becoming increasingly deeper, wider, and nonlinear due to the growing demands on prediction accuracy and analysis quality. When training a DNN model, the intermediate activation data must be saved in the memory during forward propagation and then restored for backward propagation. Traditional memory saving techniques such as data recomputation and migration either suffers from a high performance overhead or is constrained by specific interconnect technology and limited bandwidth. In this paper, we propose a novel memory-driven high performance CNN training framework that leverages error-bounded lossy compression to significantly reduce the memory requirement for training in order to allow training larger neural networks. Specifically, we provide theoretical analysis and then propose an improved lossy compressor and an adaptive scheme to dynamically configure the lossy compression error-bound and adjust the training batch size to further utilize the saved memory space for additional speedup. We evaluate our design against state-of-the-art solutions with four widely-adopted CNNs and the ImangeNet dataset. Results demonstrate that our proposed framework can significantly reduce the training memory consumption by up to 13.5× and 1.8× over the baseline training and state-of-the-art framework with compression, respectively, with little or no accuracy loss. The full paper can be referred to at https://arxiv.org/abs/2011.09017.