Trigonometric Inference Providing Learning in Deep Neural Networks
Trigonometric Inference Providing Learning in Deep Neural Networks
复制标题
DOI：
10.3390/app11156704
复制
发表时间：
2021-07
期刊：
影响因子：
--
通讯作者：
Jingyong Cai;Masashi Takemoto;Yuming Qiu;H. Nakajo
中科院分区：
文献类型：
--
作者：
Jingyong Cai;Masashi Takemoto;Yuming Qiu;H. Nakajo
尽管乘法器在深度神经网络（DNN）的训练中被大量使用，但在许多不同的场景中，乘法器都是资源密集型的。先前的发现揭示了激活函数（如sigmoid）通过移位和加法运算计算时的优越性，尽管它们无法完全消除训练中的乘法。在本文中，我们提出了一种创新的方法，可以将DNN的前向和后向推理中的所有乘法转换为移位和加法运算。由于大型DNN模型的模型参数和反向传播误差通常聚集在零附近，因此这些值可以近似为正弦值。权值和误差信号之间的乘法被转换为它们的正弦值的乘法，借助于乘积求和公式，可以用更简单的运算来代替。此外，整流正弦激活函数用于进一步将层输入转换为正弦值。通过这种方式，原始的乘法密集型运算可以通过简单的加法和移位运算来计算。这种三角近似方法为硬件乘法器不足的设备提供了一种有效的训练和推理方法。实验结果表明，该方法能够获得接近经典训练算法的性能。我们提出的方法为未来机器学习的硬件定制研究提供了新的思路。
Despite being heavily used in the training of deep neural networks (DNNs), multipliers are resource-intensive and insufficient in many different scenarios. Previous discoveries have revealed the superiority when activation functions, such as the sigmoid, are calculated by shift-and-add operations, although they fail to remove multiplications in training altogether. In this paper, we propose an innovative approach that can convert all multiplications in the forward and backward inferences of DNNs into shift-and-add operations. Because the model parameters and backpropagated errors of a large DNN model are typically clustered around zero, these values can be approximated by their sine values. Multiplications between the weights and error signals are transferred to multiplications of their sine values, which are replaceable with simpler operations with the help of the product to sum formula. In addition, a rectified sine activation function is utilized for further converting layer inputs into sine values. In this way, the original multiplication-intensive operations can be computed through simple add-and-shift operations. This trigonometric approximation method provides an efficient training and inference alternative for devices with insufficient hardware multipliers. Experimental results demonstrate that this method is able to obtain a performance close to that of classical training algorithms. The approach we propose sheds new light on future hardware customization research for machine learning.