A Contrastive Knowledge Transfer Framework for Model Compression and Transfer Learning
A Contrastive Knowledge Transfer Framework for Model Compression and Transfer Learning
复制标题
DOI：
10.1109/icassp49357.2023.10095744
复制
发表时间：
2023-03
期刊：
影响因子：
--
通讯作者：
Kaiqi Zhao;Yitao Chen;Ming Zhao
中科院分区：
文献类型：
--
作者：
Kaiqi Zhao;Yitao Chen;Ming Zhao
知识转移（KT）可实现竞争性能，并广泛用于模型压缩和转移学习中的图像分类任务。现有的KT工作将信息从大型模型（“教师”）转移到训练小型模型（“学生”）中，通过最大程度地减少其有条件独立的输出分布的差异。但是，这些作品从教师的中间表示中忽略了高维度的结构知识，这导致了有限的效果，并且是出于各种启发式直觉的动机，这使得很难概括。本文提出了一个新颖的对比知识转移框架（CKTF），该框架可以通过优化跨他们之间的中间表示的多个对比目标，从而使足够的结构知识从教师转移到学生。此外，CKTF为现有KT技术提供了广义协议，并通过将其作为CKTF的特定案例来大大提高其性能。广泛的评估表明，CKTF在模型压缩中始终胜过现有的KT的工作量为0.04％至11.59％，在各种型号和数据集上的转移学习中，CKTF的运作方式均为0.4％至4.75％。
Knowledge Transfer (KT) achieves competitive performance and is widely used for image classification tasks in model compression and transfer learning. Existing KT works transfer the information from a large model ("teacher") to train a small model ("student") by minimizing the difference of their conditionally independent output distributions. However, these works overlook the high-dimension structural knowledge from the intermediate representations of the teacher, which leads to limited effectiveness, and they are motivated by various heuristic intuitions, which makes it difficult to generalize. This paper proposes a novel Contrastive Knowledge Transfer Framework (CKTF), which enables the transfer of sufficient structural knowledge from the teacher to the student by optimizing multiple contrastive objectives across the intermediate representations between them. Also, CKTF provides a generalized agreement to existing KT techniques and increases their performance significantly by deriving them as specific cases of CKTF. The extensive evaluation shows that CKTF consistently outperforms the existing KT works by 0.04% to 11.59% in model compression and by 0.4% to 4.75% in transfer learning on various models and datasets.