Improving Robustness of Neural Networks against Bit Flipping Errors during Inference
Improving Robustness of Neural Networks against Bit Flipping Errors during Inference
复制标题
提高神经网络在推理过程中抵抗位翻转错误的鲁棒性
DOI：
10.18178/joig.6.2.181-186
复制
发表时间：
2018
影响因子：
--
通讯作者：
D. Vučinić
中科院分区：
文献类型：
--
作者：
Minghai Qin;San Jose California Usa Western Digital Coporation;Chao Sun;D. Vučinić
我们研究了存储在噪声存储介质中的神经网络的预测精度和存储冗余之间的权衡。训练后的神经网络参数通常以二进制数据的形式存储，并且通常假定数据的存储和检索是无错误的。这种假设是基于纠错码（ecc）的普遍使用，纠错码纠正存储介质中的位翻转。然而，ecc会产生容量和功率开销（10%到20%），因此在推理期间从存储中检索训练参数时增加了成本并降低了有效带宽。我们测量了几种深度神经网络架构和数据集在存在位翻转错误但在推理过程中不使用ecc时的鲁棒性。可以观察到，更复杂的架构和数据集通常更容易受到位翻转错误的影响。我们提出了一种简单的参数误差检测方法，称为权值零化，它可以根据网络结构的不同将鲁棒性从两倍提高到几个数量级。
We study the trade-offs between prediction accuracy and storage redundancy of neural networks that are stored in noisy storage media. Parameters of a trained neural network are commonly stored as binary data and it is usually assumed that the data storage and retrieval are error-free. This assumption is based upon the common use of Error Correcting Codes (ECCs) that correct bit flips in storage media. However, ECCs incur capacity and power overhead (10% to 20%) and thus increase cost and reduce the effective bandwidth when retrieving trained parameters from storage during inference. We measured the robustness of several deep neural network architectures and datasets when bit flipping errors exist but ECCs are not used during inference. It is observed that more sophisticated architectures and datasets are generally more vulnerable to bit flipping errors. We propose a simple parameter error detection method, called weight nulling, that can universally improve the robustness from twice to several orders of magnitude depending on network architectures.