Improving Deep Learning with Differential Privacy using Gradient Encoding and Denoising
Improving Deep Learning with Differential Privacy using Gradient Encoding and Denoising
复制标题
DOI：
--
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Milad Nasr;R. Shokri;Amir Houmansadr
中科院分区：
文献类型：
--
作者：
Milad Nasr;R. Shokri;Amir Houmansadr
深度学习模型会泄露大量关于其训练数据集的信息。以前的工作是通过在梯度中加入差分隐私噪声来研究差分隐私（DP）保证的训练模型。然而，这样的解决方案（特别是DPSGD）会导致训练模型的准确性大大降低。在本文中，我们的目标是训练具有DP保证的深度学习模型，同时保持比以前的工作更好的模型准确性。我们的关键技术是对梯度进行编码，将它们映射到更小的向量空间，从而使我们能够获得不同噪声分布的DP保证。这允许我们调查和选择噪声分布，以最好地保持目标隐私预算的模型准确性。我们还通过引入去噪的思想来利用差分隐私的后处理特性，这进一步提高了训练模型的效用，而不会降低其DP保证。我们表明，我们的机制优于最先进的DPSGD；例如，在MNIST上相同的模型精度为$96.1\%$时，我们的技术产生的隐私界为$\epsilon=3.2$，而DPSGD的隐私界为$\epsilon=6$，这是一个显著的改进。
Deep learning models leak significant amounts of information about their training datasets. Previous work has investigated training models with differential privacy (DP) guarantees through adding DP noise to the gradients. However, such solutions (specifically, DPSGD), result in large degradations in the accuracy of the trained models. In this paper, we aim at training deep learning models with DP guarantees while preserving model accuracy much better than previous works. Our key technique is to encode gradients to map them to a smaller vector space, therefore enabling us to obtain DP guarantees for different noise distributions. This allows us to investigate and choose noise distributions that best preserve model accuracy for a target privacy budget. We also take advantage of the post-processing property of differential privacy by introducing the idea of denoising, which further improves the utility of the trained models without degrading their DP guarantees. We show that our mechanism outperforms the state-of-the-art DPSGD; for instance, for the same model accuracy of $96.1\%$ on MNIST, our technique results in a privacy bound of $\epsilon=3.2$ compared to $\epsilon=6$ of DPSGD, which is a significant improvement.