Practical Differentially Private and Byzantine-resilient Federated Learning
Practical Differentially Private and Byzantine-resilient Federated Learning
复制标题
DOI：
10.1145/3589264
复制
发表时间：
2023-04
期刊：
影响因子：
--
通讯作者：
Zihang Xiang;Tianhao Wang;Wanyu Lin;Di Wang
中科院分区：
文献类型：
--
作者：
Zihang Xiang;Tianhao Wang;Wanyu Lin;Di Wang
隐私和拜占庭式的弹性是联邦学习（FL）系统不可或缺的两个要求。虽然在隐私和拜占庭安全方面已经有了广泛的研究，但考虑到这两个方面的解决方案仍然很少。这是由于难以协调隐私保护和拜占庭弹性算法。在这项工作中，我们提出了这样一个双重问题的解决方案。我们使用我们的差分私有随机梯度下降（DP-SGD）算法来保护隐私，然后应用我们的拜占庭弹性算法。我们注意到，虽然现有的作品遵循这种一般的方法，DP和拜占庭弹性之间的相互作用的深入分析已被忽略，导致不满意的性能。具体来说，对于DP引入的随机噪声，以前的工作努力减少其看似有害的影响拜占庭聚集。相比之下，我们利用随机噪声来构建第一阶段聚合，有效地拒绝了许多现有的拜占庭攻击。此外，基于我们的DP变体的另一个属性，我们形成了提供最终声音过滤的第二阶段聚合。我们的协议遵循共同设计DP和拜占庭弹性的原则。我们提供了理论证明和实证实验来证明我们的协议是有效的：保持高准确性，同时保持DP保证和拜占庭弹性。与以前的工作相比，我们的协议1）即使在高隐私制度下也能达到更高的准确性; 2）即使高达90%的分布式工作者是拜占庭式的，也能很好地工作。
Privacy and Byzantine resilience are two indispensable requirements for a federated learning (FL) system. Although there have been extensive studies on privacy and Byzantine security in their own track, solutions that consider both remain sparse. This is due to difficulties in reconciling privacy-preserving and Byzantine-resilient algorithms. In this work, we propose a solution to such a two-fold issue. We use our version of differentially private stochastic gradient descent (DP-SGD) algorithm to preserve privacy and then apply our Byzantine-resilient algorithms. We note that while existing works follow this general approach, an in-depth analysis on the interplay between DP and Byzantine resilience has been ignored, leading to unsatisfactory performance. Specifically, for the random noise introduced by DP, previous works strive to reduce its seemingly detrimental impact on the Byzantine aggregation. In contrast, we leverage the random noise to construct a first-stage aggregation that effectively rejects many existing Byzantine attacks. Moreover, based on another property of our DP variant, we form a second-stage aggregation which provides a final sound filtering. Our protocol follows the principle of co-designing both DP and Byzantine resilience. We provide both theoretical proof and empirical experiments to show our protocol is effective: retaining high accuracy while preserving the DP guarantee and Byzantine resilience. Compared with the previous work, our protocol 1) achieves significantly higher accuracy even in a high privacy regime; 2) works well even when up to 90% distributive workers are Byzantine.