Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clipping
Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clipping
复制标题
DOI：
10.2478/popets-2021-0008
复制
发表时间：
2020-09
影响因子：
--
通讯作者：
Jaewoo Lee;Daniel Kifer
中科院分区：
文献类型：
--
作者：
Jaewoo Lee;Daniel Kifer
关于Renyi差异隐私的最新工作表明，尽管有希望，将差异性隐私应用于深度学习，但私人的深层网络通常落后于其准确性的非私有网络，表明对模型架构，优化者等更多的障碍，比训练时间更大这种放缓的原因是与隐私相关的步骤，称为“每个例子梯度剪辑”，其天真的实现通过分析后反射方程来解散批处理培训的好处，我们为每个示例梯度的新方法提供了与自动剪辑相兼容的新方法。大量训练的速度（通过54x -94倍用于培训各种批量的128个模型）。
Abstract Recent work on Renyi Differential Privacy has shown the feasibility of applying differential privacy to deep learning tasks. Despite their promise, however, differentially private deep networks often lag far behind their non-private counterparts in accuracy, showing the need for more research in model architectures, optimizers, etc. One of the barriers to this expanded research is the training time — often orders of magnitude larger than training non-private networks. The reason for this slowdown is a crucial privacy-related step called “per-example gradient clipping” whose naive implementation undoes the benefits of batch training with GPUs. By analyzing the back-propagation equations we derive new methods for per-example gradient clipping that are compatible with auto-differeniation (e.g., in Py-Torch and TensorFlow) and provide better GPU utilization. Our implementation in PyTorch showed significant training speed-ups (by factors of 54x - 94x for training various models with batch sizes of 128). These techniques work for a variety of architectural choices including convolutional layers, recurrent networks, attention, residual blocks, etc.