Hunter: HE-Friendly Structured Pruning for Efficient Privacy-Preserving Deep Learning
Hunter: HE-Friendly Structured Pruning for Efficient Privacy-Preserving Deep Learning
复制标题
DOI：
10.1145/3488932.3517401
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Yifei Cai;Qiao Zhang;R. Ning;Chunsheng Xin;Hongyi Wu
中科院分区：
文献类型：
--
作者：
Yifei Cai;Qiao Zhang;R. Ning;Chunsheng Xin;Hongyi Wu
为了在机器学习即服务(MLaaS)中保护用户隐私，人们提出了一系列设计巧妙的隐私保护框架。最新的方法对线性函数采用同态加密(HE)，对非线性运算采用乱码电路(GC)/不经意转移(OT)，以提高计算效率。尽管取得了令人鼓舞的进展，但对于实际应用来说，计算成本仍然太高。这项工作是有效修剪隐私保护深度学习模型以降低计算复杂性的第一步。虽然模型剪枝在机器学习领域已经得到了广泛的讨论，但是直接应用明文模型剪枝方案对减少隐私保护模型的计算量帮助不大。在本文中，我们提出了一种结构化剪枝方法Hunter，它识别了三种新的HE友好结构，即内部结构、外部结构和权重对角线，以指导剪枝过程。Hunter输出的剪枝模型在不损失模型精度的情况下，在隐私保护的MLaaS中显著减少了HE操作(从而降低了总体计算成本)。我们将Hunter应用于各种深度学习模型，例如AlexNet、VGG和ResNet，并在包括MNIST、CIFAR-10和ImageNet在内的经典数据集上进行了应用。实验结果表明，在不损失精度的情况下，Hunter有效地修剪了原始网络，减少了HE Perm、Mult和Add运算。例如，在ImageNet上最先进的VGG-16中，有10个选择的类，PERM的总数减少到原始网络的2%，同时MULT和ADD减少到只有14%，从而实现了显著更高效的计算效率和隐私保护MLaaS。
In order to protect user privacy in Machine Learning as a Service (MLaaS), a series of ingeniously designed privacy-preserving frameworks have been proposed. The state-of-the-art approaches adopt Homomorphic Encryption (HE) for linear function and Garbled Circuits (GC)/Oblivious Transfer (OT) for nonlinear operation to improve computation efficiency. Despite the encouraging progress, the computation cost is still too high for practical applications. This work represents the first step to effectively prune privacy-preserving deep learning models to reduce computation complexity. Although model pruning has been discussed extensively in the machine learning community, directly applying the plaintext model pruning schemes offers little help to reduce the computation in privacy-preserving models. In this paper we propose Hunter, a structured pruning method that identifies three novel HE-friendly structures, i.e., internal structure, external structure, and weight diagonal to guide the pruning process. Hunter outputs a pruned model that, without any loss in model accuracy, achieves a significant reduction in HE operations (and thus the overall computation cost) in the privacy-preserving MLaaS. We apply Hunter in various deep learning models, e.g., AlexNet, VGG and ResNet over classic datasets including MNIST, CIFAR-10 and ImageNet. The experimental results demonstrate that, without accuracy loss, Hunter efficiently prunes the original networks to reduce the HE Perm, Mult, and Add operations. For example, in the state-of-the-art VGG-16 on ImageNet with 10 chosen classes, the total number of Perm is reduced to as low as 2% of the original network, and at the same time, Mult and Add are reduced to only 14%, enabling a significantly more computation-efficient privacy-preserving MLaaS.