ESRU: Extremely Low-Bit and Hardware-Efficient Stochastic Rounding Unit Design for Low-Bit DNN Training
ESRU: Extremely Low-Bit and Hardware-Efficient Stochastic Rounding Unit Design for Low-Bit DNN Training
复制标题
DOI：
10.23919/date56975.2023.10137222
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Sung-En Chang;Geng Yuan;Alec Lu;Mengshu Sun;Yanyu Li;Xiaolong Ma;Z. Li;Yanyue Xie;Minghai Qin;Xue Lin;Zhenman Fang;Yanzhi Wang
中科院分区：
文献类型：
--
作者：
Sung-En Chang;Geng Yuan;Alec Lu;Mengshu Sun;Yanyu Li;Xiaolong Ma;Z. Li;Yanyue Xie;Minghai Qin;Xue Lin;Zhenman Fang;Yanzhi Wang
随机舍入在低位中是至关重要的（例如，8位）训练深度神经网络（DNN），以实现高精度。现有研究的缺点之一是，它们需要大量的高精度随机舍入单元（SRU）来保证低位DNN精度，这涉及相当大的硬件开销。在本文中，我们使用极低比特的SRU（ESRU）来节省大量的硬件资源在低比特DNN训练。然而，一个简单设计的ESRU引入了随机数的有偏分布，导致精度下降。为了解决这个问题，我们进一步提出了一个ESRU的设计与平台形状的分布。在我们的ESRU设计中的平台形分布实现了一个LFSR（线性反馈移位寄存器）和一个反相LFSR的组合，这避免了LFSR包装，并把一个固有的LFSR缺点变成一个优势，在我们的高效ESRU设计。使用最先进的DNN模型的实验结果表明，与之前的24位SRU与24位伪随机数生成器（PRNG）相比，我们的8位ESRU与3位PRNG将SRU硬件资源使用量减少了9.75倍，同时实现了略高的精度。
Stochastic rounding is crucial in the low-bit (e.g., 8-bit) training of deep neural networks (DNNs) to achieve high accuracy. One of the drawbacks of prior studies is that they require a large number of high-precision stochastic rounding units (SRUs) to guarantee low-bit DNN accuracy, which involves considerable hardware overhead. In this paper, we use extremely low-bit SRUs (ESRUs) to save a large number of hardware resources during low-bit DNN training. However, a naively designed ESRU introduces a biased distribution of random numbers, causing accuracy degradation. To address this issue, we further propose an ESRU design with a plateau-shape distribution. The plateau-shape distribution in our ESRU design is implemented with the combination of an LFSR (linear-feedback shift register) and an inverted LFSR, which avoids LFSR packing and turns an inherent LFSR drawback into an advantage in our efficient ESRU design. Experimental results using state-of-the-art DNN models demonstrate that, compared to the prior 24-bit SRU with 24-bit pseudo-random number generators (PRNG), our 8-bit ESRU with 3-bit PRNG reduces the SRU hardware resource usage by 9.75x while achieving slightly higher accuracy.