Analyzing the Confidentiality of Undistillable Teachers in Knowledge Distillation
Analyzing the Confidentiality of Undistillable Teachers in Knowledge Distillation
复制标题
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Souvik Kundu;Qirui Sun;Yao Fu;M. Pedram;P. Beerel
中科院分区：
文献类型：
--
作者：
Souvik Kundu;Qirui Sun;Yao Fu;M. Pedram;P. Beerel
知识蒸馏（KD）最近被艾德为一种可能无意中向未经授权的学生泄露有关教师模型细节的私人信息的方法。最近的一项研究是开发能够保护模型机密性的不可否认的讨厌的教师，这一研究引起了人们的极大关注。然而，这些讨厌的模型提供的保护水平在很大程度上未经测试。在本文中，我们表明，知识转移到一个浅的学生的子部分可以在很大程度上减少教师的影响力。通过探索浅层子部分的深度，我们提出了一种蒸馏技术，使一个持怀疑态度的学生模型，甚至从一个讨厌的老师学习。为了评估我们持怀疑态度的学生的有效性，我们在各种数据集的训练数据可用和无数据场景下使用KD进行了几个模型的实验。与正常学生模型相比，怀疑型学生的分类成绩一直保持在上级水平，最高可达159分。百分之五此外，与正常学生类似，当从正常教师中提取时，持怀疑态度的学生保持了很高的分类准确性，无论教师是否讨厌，他们都表现出了效率。我们相信，持怀疑态度的学生能够在很大程度上削弱一个潜在的讨厌的老师的KD免疫力，这将激励研究界为模型确认创建更强大的机制。我们在github.com/ksouvik52/Skeptical2021上开源了代码。
Knowledge distillation (KD) has recently been identified as a method that can unintentionally leak private information regarding the details of a teacher model to an unauthorized student. Recent research in developing undistillable nasty teachers that can protect model confidentiality has gained significant attention. However, the level of protection these nasty models offer has been largely untested. In this paper, we show that transferring knowledge to a shallow sub-section of a student can largely reduce a teacher’s influence. By exploring the depth of the shallow subsection, we then present a distillation technique that enables a skeptical student model to learn even from a nasty teacher. To evaluate the efficacy of our skeptical students, we conducted experiments with several models with KD under both training data-available and data-free scenarios for various datasets. While distilling from nasty teachers, compared to the normal student models, skeptical students consistently provide superior classification performance of up to ∼ 59 . 5% . Moreover, similar to normal students, skeptical students maintain high classification accuracy when distilled from a normal teacher, showing their efficacy irrespective of the teacher being nasty or not. We believe the ability of skeptical students to largely diminish the KD-immunity of a potentially nasty teacher will motivate the research community to create more robust mechanisms for model confidentiality. We have open-sourced the code at github.com/ksouvik52/Skeptical2021 .