You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding-猫眼课题宝

DOI：

10.1007/978-3-031-19775-8_3

发表时间：

2022

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Geng Yuan;Sung-En Chang;Qing Jin;Alec Lu;Yanyu Li;Yushu Wu;Zhenglun Kong;Yanyue Xie;Peiyan Dong;Minghai Qin;Xiaolong Ma;Xulong Tang;Zhenman Fang;Yanzhi Wang

随机舍入是用于低精度深度神经网络（DNN）训练的关键技术，以确保良好的模型精度。然而，它需要在运行中生成大量的随机数。在FPGA和ASIC等硬件平台上，这不是一个简单的任务。广泛使用的解决方案是引入具有额外硬件成本的随机数生成器。在本文中，我们创新性地提出利用DNN训练过程本身的随机性，以自给自足的方式直接从DNN中提取随机数。我们提出了不同的方法来从神经网络中的不同来源获得随机数，并提出了一个无生成器的框架，用于各种深度学习任务的低精度DNN训练。此外，我们评估了提取的随机数的质量，发现高质量的随机数广泛存在于DNN中，而它们的质量甚至可以通过NIST测试套件。

Stochastic rounding is a critical technique used in low-precision deep neural networks (DNNs) training to ensure good model accuracy. However, it requires a large number of random numbers generated on the fly. This is not a trivial task on the hardware platforms such as FPGA and ASIC. The widely used solution is to introduce random number generators with extra hardware costs. In this paper, we innovatively propose to employ the stochastic property of DNN training process itself and directly extract random numbers from DNNs in a self-sufficient manner. We propose different methods to obtain random numbers from different sources in neural networks and a generator-free framework is proposed for low-precision DNN training on a variety of deep learning tasks. Moreover, we evaluate the quality of the extracted random numbers and find that high-quality random numbers widely exist in DNNs, while their quality can even pass the NIST test suite.

You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding

文献摘要

相似文献