Learning to Explore Distillability and Sparsability: A Joint Framework for Model Compression
Learning to Explore Distillability and Sparsability: A Joint Framework for Model Compression
复制标题
学习探索可蒸馏性和可稀疏性：模型压缩的联合框架
DOI：
10.1109/tpami.2022.3185317
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Stephen Maybank
中科院分区：
文献类型：
--
作者：
Yufan Liu;Jiajiong Cao;Bing Li;Weiming Hu;Stephen Maybank
Deep learning shows excellent performance usually at the expense of heavy computation. Recently, model compression has become a popular way of reducing the computation. Compression can be achieved using knowledge distillation or filter pruning. Knowledge distillation improves the accuracy of a lightweight network, while filter pruning removes redundant architecture in a cumbersome network. They are two different ways of achieving model compression, but few methods simultaneously consider both of them. In this paper, we revisit model compression and define two attributes of a model: distillability and sparsability, which reflect how much useful knowledge can be distilled and how many pruned ratios can be obtained, respectively. Guided by our observations and considering both accuracy and model size, a dynamically distillability-and-sparsability learning framework (DDSL) is introduced for model compression. DDSL consists of teacher, student and dean. Knowledge is distilled from the teacher to guide the student. The dean controls the training process by dynamically adjusting the distillation supervision and the sparsity supervision in a meta-learning framework. An alternating direction method of multiplier (ADMM)-based knowledge distillation-with-pruning (KDP) joint optimization algorithm is proposed to train the model. Extensive experimental results show that DDSL outperforms 24 state-of-the-art methods, including both knowledge distillation and filter pruning methods.
登录
查看更多内容
DOI：
10.1609/aaai.v35i9.16969
发表时间：
2021-02
期刊：
ArXiv
影响因子：
--
作者：
Mingi Ji;Byeongho Heo;Sungrae Park
通讯作者：
Mingi Ji;Byeongho Heo;Sungrae Park
DOI：
10.1109/cvpr.2018.00171
发表时间：
2017-11
期刊：
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
影响因子：
--
作者：
A. Gordon;Elad Eban;Ofir Nachum;Bo Chen;Tien-Ju Yang;E. Choi
通讯作者：
A. Gordon;Elad Eban;Ofir Nachum;Bo Chen;Tien-Ju Yang;E. Choi
DOI：
10.1109/cvpr.2019.01146
发表时间：
2018-12
期刊：
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
影响因子：
--
作者：
Haichuan Yang;Yuhao Zhu;Ji Liu
通讯作者：
Haichuan Yang;Yuhao Zhu;Ji Liu
DOI：
10.1609/aaai.v34i04.5924
发表时间：
2019-07
期刊：
--
影响因子：
--
作者：
Ning Liu;Xiaolong Ma;Zhiyuan Xu;Yanzhi Wang;Jian Tang;Jieping Ye
通讯作者：
Ning Liu;Xiaolong Ma;Zhiyuan Xu;Yanzhi Wang;Jian Tang;Jieping Ye
DOI：
10.1007/978-3-030-58580-8_35
发表时间：
2020-04
期刊：
ArXiv
影响因子：
--
作者：
Xuefei Ning;Tianchen Zhao;Wenshuo Li;Peng Lei;Yu Wang;Huazhong Yang
通讯作者：
Xuefei Ning;Tianchen Zhao;Wenshuo Li;Peng Lei;Yu Wang;Huazhong Yang