Compressing Neural Networks with the Hashing Trick
Compressing Neural Networks with the Hashing Trick
复制标题
DOI：
--
复制
发表时间：
2015-04
期刊：
影响因子：
--
通讯作者：
Wenlin Chen-;James T. Wilson;Stephen Tyree;Kilian Q. Weinberger;Yixin Chen
中科院分区：
文献类型：
--
作者：
Wenlin Chen-;James T. Wilson;Stephen Tyree;Kilian Q. Weinberger;Yixin Chen
随着深度网络越来越多地用于适合移动的设备的应用中，一个基本的困境变得明显：深度学习的趋势是增长模型以吸收不断增加的数据集大小;然而，移动的设备被设计为具有非常小的内存，并且不能存储如此大的模型。我们提出了一种新的网络架构，哈希网，利用神经网络中固有的冗余，以实现模型大小的大幅减少。HashedNets使用低成本的哈希函数将连接权重随机分组到哈希桶中，同一哈希桶中的所有连接共享一个参数值。这些参数经过调整，以适应训练期间具有标准反向传播的HashedNets权重共享架构。我们的散列过程不会引入额外的内存开销，并且我们在几个基准数据集上证明了HashedNets大幅缩减了神经网络的存储需求，同时大部分保留了泛化性能。
As deep nets are increasingly used in applications suited for mobile devices, a fundamental dilemma becomes apparent: the trend in deep learning is to grow models to absorb ever-increasing data set sizes; however mobile devices are designed with very little memory and cannot store such large models. We present a novel network architecture, HashedNets, that exploits inherent redundancy in neural networks to achieve drastic reductions in model sizes. HashedNets uses a low-cost hash function to randomly group connection weights into hash buckets, and all connections within the same hash bucket share a single parameter value. These parameters are tuned to adjust to the HashedNets weight sharing architecture with standard backprop during training. Our hashing procedure introduces no additional memory overhead, and we demonstrate on several benchmark data sets that HashedNets shrink the storage requirements of neural networks substantially while mostly preserving generalization performance.