Recurrent Residual Networks Contain Stronger Lottery Tickets
Recurrent Residual Networks Contain Stronger Lottery Tickets
复制标题
DOI：
10.1109/access.2023.3245808
复制
发表时间：
2023
期刊：
影响因子：
3.9
通讯作者：
Ángel López García-Arias;Yasuyuki Okoshi;Masanori Hashimoto;Masato Motomura;Jaehoon Yu
中科院分区：
文献类型：
--
作者：
Ángel López García-Arias;Yasuyuki Okoshi;Masanori Hashimoto;Masato Motomura;Jaehoon Yu
只需对随机初始化的超参数模型进行剪枝，就可以找到精确的神经网络，而不需要进行任何权重优化。由此产生的子网络很小、稀疏和三元，这是高效硬件实施的极佳候选者。然而，寻找最佳连接模式是一项尚未解决的挑战。基于残差网络可能逼近展开的浅层递归神经网络的证据，我们推测，当显式变换为递归结构时，它们在推理时包含更好的候选子网络。将这一假设用于图像分类任务，我们在递归模型中找到比前馈模型中的子网络更准确、参数效率更高的子网络，并且比具有学习权重的完整模型更准确和更有效。此外，随机递归子网络很小：在一个简单的压缩方案下，ResNet-50在没有严重性能损失的情况下被压缩到$48.55\x$x内存大小，适合不到2兆字节。代码请访问：https://github.com/Lopez-Angel/hidden-fold-networks.
Accurate neural networks can be found just by pruning a randomly initialized overparameterized model, leaving out the need for any weight optimization. The resulting subnetworks are small, sparse, and ternary, making excellent candidates for efficient hardware implementation. However, finding optimal connectivity patterns is an open challenge. Based on the evidence that residual networks may be approximating unrolled shallow recurrent neural networks, we conjecture that they contain better candidate subnetworks at inference time when explicitly transformed into recurrent architectures. This hypothesis is put to the test on image classification tasks, where we find subnetworks within the recurrent models that are more accurate and parameter-efficient than both the ones found within feedforward models and than the full models with learned weights. Furthermore, random recurrent subnetworks are tiny: under a simple compression scheme, ResNet-50 is compressed without a drastic loss in performance to $48.55\times $ less memory size, fitting in under 2 megabytes. Code available at: https://github.com/Lopez-Angel/hidden-fold-networks.