Scalable Differential Privacy with Sparse Network Finetuning
Scalable Differential Privacy with Sparse Network Finetuning
复制标题
DOI：
10.1109/cvpr46437.2021.00502
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Zelun Luo;Daniel J. Wu;E. Adeli;L. Fei-Fei-L.-Fei-Fei-1435579960
中科院分区：
文献类型：
--
作者：
Zelun Luo;Daniel J. Wu;E. Adeli;L. Fei-Fei-L.-Fei-Fei-1435579960
我们提出了一种利用公共域外数据对深度神经网络进行隐私保护训练的新方法。虽然差分隐私（DP）已经成为一种保护训练数据集中敏感数据的机制，但其在复杂视觉识别任务中的应用仍然具有挑战性。传统的DP方法，如差分私有随机梯度下降（DP-SGD），仅在简单数据集和浅网络上表现良好，而最近基于迁移学习的DP方法通常对公共数据的可用性和分布做出不切实际的假设。在这项工作中，我们认为，最大限度地减少可训练参数的数量是提高DP在复杂的视觉识别任务上的隐私性能权衡的关键。受这一论点的启发，我们还提出了一种新的迁移学习范式，用DP微调一个非常稀疏的子网络。我们对两个视觉识别任务进行了广泛的实验和消融研究：CIFAR-100 → CIFAR-10（标准DP设置）和CD-FSL挑战（少数镜头，多层次的域转移），并展示了具有竞争力的实验性能。
We propose a novel method for privacy-preserving training of deep neural networks leveraging public, out-domain data. While differential privacy (DP) has emerged as a mechanism to protect sensitive data in training datasets, its application to complex visual recognition tasks remains challenging. Traditional DP methods, such as Differentially-Private Stochastic Gradient Descent (DP-SGD), perform well only on simple datasets and shallow networks, while recent transfer learning-based DP methods often make unrealistic assumptions about the availability and distribution of public data. In this work, we argue that minimizing the number of trainable parameters is the key to improving the privacy-performance tradeoff of DP on complex visual recognition tasks. Inspired by this argument, we also propose a novel transfer learning paradigm that finetunes a very sparse subnetwork with DP. We conduct extensive experiments and ablation studies on two visual recognition tasks: CIFAR-100 → CIFAR-10 (standard DP setting) and the CD-FSL challenge (few-shot, multiple levels of domain shifts) and demonstrate competitive experimental performance.