Building and road detection from remote sensing images based on weights adaptive multi-teacher collaborative distillation using a fused knowledge
Building and road detection from remote sensing images based on weights adaptive multi-teacher collaborative distillation using a fused knowledge
复制标题
DOI：
10.1016/j.jag.2023.103522
复制
发表时间：
2023-11
期刊：
影响因子：
--
通讯作者：
Zi-xing Chen;Liai Deng;Jing Gou;Cheng Wang;Jonathan Li;Dilong Li
中科院分区：
文献类型：
--
作者：
Zi-xing Chen;Liai Deng;Jing Gou;Cheng Wang;Jonathan Li;Dilong Li
知识蒸馏是压缩深度学习模型的一种有效方法。然而，目前的蒸馏方法相对单一。目前，关于使用多种知识类型和采用多种教师模型的蒸馏策略组合的研究还很少。此外，如何优化不同教师模型之间的权重仍然是一个悬而未决的问题。针对这些问题，提出了一种新的知识蒸馏方法，通过权重自适应的多教师协同蒸馏，有效地提高了蒸馏学生模型的鲁棒性。此外，该方法利用教师网络之间的特征知识交换指导，将更全面的特征知识传递给学生模型，进一步提高了隐藏层细节的学习能力。大量的实验结果表明，所提出的方法达到了最先进的性能在马萨诸塞州道路数据集，LRSNY道路数据集，WHU建筑数据集。具体而言，在第一个教师网络集成的指导下，我们分别获得了47.33%，78.15%和80.71%的IoU分数。在第二次教师网络集成的指导下，我们分别获得了48.56%，79.51%和81.35%的IoU得分。
Knowledge distillation is one effective approach to compress deep learning models. However, the current distillation methods are relatively monotonous. There are still rare studies about the combination of distillation strategies using multiple types of knowledge and employing multiple teacher models. Besides, how to optimize the weights among different teacher models is still an open problem. To address these issues, this paper proposes a novel approach for knowledge distillation, which effectively enhances the robustness of the distilled student model by a weights adaptive multi-teacher collaborative distillation. Moreover, the proposed method utilizes feature knowledge exchange guidance between teacher networks to transfer more comprehensive feature knowledge to the student model, which further improves the learning capability of hidden layers’ details. The extensive experimental results demonstrate that the proposed method achieves state-of-the-art performance on Massachusetts Roads Dataset, LRSNY Roads Dataset, and WHU Building Dataset. Specifically, under the guidance of the first ensemble of teacher networks, we obtained IoU scores of 47.33%, 78.15%, and 80.71%, respectively. Under the guidance of the second ensemble of teacher networks, we obtained IoU scores of 48.56%, 79.51%, and 81.35%, respectively.