Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learning
Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learning
复制标题
DOI：
10.1109/dac56929.2023.10247920
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Y. Fu;Ye Yuan;Shang Wu;Jiayi Yuan;Yingyan Lin
中科院分区：
文献类型：
--
作者：
Y. Fu;Ye Yuan;Shang Wu;Jiayi Yuan;Yingyan Lin
转移学习利用在具有丰富数据的源任务上预先训练的深度神经网络(DNN)的特征表示来支持对下游任务的有效微调。然而，预先训练的模型通常大得令人望而却步，无法提供可概括的表示，这限制了它们在资源有限的边缘设备上的部署。为了缩小这一差距，我们提出了一种新的转移学习管道，该管道利用了我们的发现，稳健的彩票可以更好地转移，即，具有适当诱导的对抗性健壮性的子网络可以比普通彩票子网络赢得更好的可转移性。大量的实验和消融研究证明，我们提出的转移学习流水线可以在不同的下游任务和稀疏模式之间实现更高的精度-稀疏性权衡，进一步丰富了彩票假说。
Transfer learning leverages feature representations of deep neural networks (DNNs) pretrained on source tasks with rich data to empower effective finetuning on downstream tasks. However, the pre-trained models are often prohibitively large for delivering generalizable representations, which limits their deployment on edge devices with constrained resources. To close this gap, we propose a new transfer learning pipeline, which leverages our finding that robust tickets can transfer better, i.e., subnetworks drawn with properly induced adversarial robustness can win better transferability over vanilla lottery ticket subnetworks. Extensive experiments and ablation studies validate that our proposed transfer learning pipeline can achieve enhanced accuracy-sparsity trade-offs across both diverse downstream tasks and sparsity patterns, further enriching the lottery ticket hypothesis.