Energy Efficient Learning With Low Resolution Stochastic Domain Wall Synapse for Deep Neural Networks
Energy Efficient Learning With Low Resolution Stochastic Domain Wall Synapse for Deep Neural Networks
复制标题
用于深度神经网络的低分辨率随机畴壁突触的节能学习
DOI：
10.1109/access.2022.3196688
复制
发表时间：
2022
期刊：
影响因子：
3.9
通讯作者：
Atulasimha, Jayasimha
中科院分区：
文献类型：
--
作者：
Misba, Walid Al;Lozano, Mark;Querlioz, Damien;Atulasimha, Jayasimha
我们证明了极低分辨率量化（标称5状态）突触与突触权重的大随机变化可以是能量有效的，并实现合理的高测试精度相比，深度神经网络（DNN）的类似大小使用浮点精度突触权重。具体来说，压控畴壁（DW）器件表现出随机行为，只能编码有限的状态;然而，它们在训练和推理过程中都非常节能。在这项研究中，我们提出了原位和异位训练算法，基于Hubaraet等人提出的算法的修改，2017年，它与突触权重的量化工作良好，并使用2-，3-和5-状态DW设备作为突触在MNIST数据集上训练几个5层DNN。对于原位训练，单独的高精度存储器单元保存并累积权重梯度，这防止了由于权重量化而导致的精度损失。对于非原位训练，首先基于权重量化和DW设备模型训练前体DNN。此外，噪声容限容限被包括在两种训练方法中以考虑固有器件噪声。我们在原位和非原位训练后获得的最高推理准确率分别为~ 96.67%和~ 96.63%，这非常接近从具有浮点精度权重且无随机性的类似拓扑DNN获得的~97.1%的基线准确率。由于量化的权重和噪声容限而产生的大的状态间间隔使得能够以显著更少的编程尝试次数进行原位训练。我们所提出的方法表明，至少两个数量级的能源节约的可能性相比，在CMOS实现的浮点方法。这种方法对于低功率智能边缘设备特别有吸引力，其中非原位学习可以用于节能的非自适应任务，而原位学习可以提供在动态演变的环境中适应和学习的机会。
We demonstrate extremely low resolution quantized (nominally 5-state) synapses with large stochastic variations in synaptic weights can be energy efficient and achieve reasonably high testing accuracies compared to Deep Neural Networks (DNNs) of similar sizes using floating-point precision synaptic weights. Specifically, voltage-controlled domain wall (DW) devices demonstrate stochastic behavior and can only encode limited states; however, they are extremely energy efficient during both training and inference. In this study, we propose both in-situ and ex-situ training algorithms, based on modification of the algorithm proposed by Hubaraet al., 2017 which works well with quantization of synaptic weights, and train several 5-layer DNNs on MNIST dataset using 2-, 3- and 5-state DW devices as a synapse. For in-situ training, a separate high precision memory unit preserves and accumulates the weight gradients which prevents accuracy loss due to weight quantization. For ex-situ training, a precursor DNN is first trained based on weight quantization and DW device model. Moreover, a noise tolerance margin is included in both of the training methods to account for the intrinsic device noise. The highest inference accuracies we obtain after in-situ and ex-situ training are ~ 96.67% and ~96.63%, respectively, which is very close to the baseline accuracy of ~97.1% obtained from a similar topology DNN having floating-point precision weights with no stochasticity. Large inter-state intervals due to quantized weights and noise tolerance margin enables in-situ training with significantly lower number of programming attempts. Our proposed approach demonstrates a possibility of at leasttwo orders of magnitude energy savings compared to the floating-point approach implemented in CMOS. This approach is specifically attractive for low power intelligent edge devices where the ex-situ learning can be utilized for energy efficient non-adaptive tasks and the in-situ learning can provide the opportunity to adapt and learn in a dynamically evolving environment.
登录
查看更多内容
影响因子：
4
作者：
Alamdar, Mahshid;Leonard, Thomas;Incorvia, Jean Anne C.
通讯作者：
Incorvia, Jean Anne C.
影响因子：
4.3
作者：
Nandakumar, S. R.;Le Gallo, Manuel;Eleftheriou, Evangelos
通讯作者：
Eleftheriou, Evangelos
影响因子：
1.6
作者：
S. Dutta;Saima A. Siddiqui;J. A. Currivan-Incorvia;C. Ross;M. Baldo
通讯作者：
S. Dutta;Saima A. Siddiqui;J. A. Currivan-Incorvia;C. Ross;M. Baldo
基于忆阻器的神经网络的离线训练
DOI：
10.23919/eusipco47968.2020.9287574
发表时间：
2021
期刊：
2020 28th European Signal Processing Conference (EUSIPCO)
影响因子：
--
作者：
Guillem Boquet;Edwar Macias Toro;A. Morell;Javier Serrano;E. Miranda;J. Vicario
通讯作者：
J. Vicario
基于忆阻器的自动编码器用于无监督实时网络入侵和异常检测
DOI：
10.1145/3354265.3354267
发表时间：
2019
期刊：
ICONS '19: Proceedings of the International Conference on Neuromorphic Systems
影响因子：
--
作者：
Md. Shahanur Alam, B. Rasitha
通讯作者：
Md. Shahanur Alam, B. Rasitha