Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks
Federated Variance-Reduced Stochastic Gradient Descent With Robustness to Byzantine Attacks
复制标题
DOI：
10.1109/tsp.2020.3012952
复制
发表时间：
2019-12
影响因子：
5.4
通讯作者：
Zhaoxian Wu;Qing Ling;Tianyi Chen;G. Giannakis
中科院分区：
文献类型：
--
作者：
Zhaoxian Wu;Qing Ling;Tianyi Chen;G. Giannakis
研究了存在恶意拜占庭攻击情况下的分布式有限和多工作者学习优化问题。到目前为止，最具弹性的方法将随机梯度下降(SGD)与不同的稳健聚集规则相结合。然而，SGD引起的相当大的随机梯度噪声挑战着将拜占庭攻击者发送的恶意消息与由诚实的工人发送的噪声随机梯度区分开来。这促使减少随机梯度的方差，以此作为增强SGD的一种手段。为此，针对涉及多个工作者的联合学习任务，提出了一种新的拜占庭攻击弹性分布式SAGA方法。与分布式佐贺所使用的平均值不同，小说伯德-佐贺依赖几何中位数来聚合工人发送的校正后的随机梯度。当不到一半的工人是拜占庭攻击者时，Byrd-Saga算法可证线性收敛到最优解的邻域，其渐近学习误差由拜占庭工人的数量决定。数值实验证实了该算法对各种拜占庭攻击的健壮性，以及该算法相对于拜占庭攻击弹性分布式SGD算法的优点。
This paper deals with distributed finite-sum optimization for learning over multiple workers in the presence of malicious Byzantine attacks. Most resilient approaches so far combine stochastic gradient descent (SGD) with different robust aggregation rules. However, the sizeable SGD-induced stochastic gradient noise challenges discerning malicious messages sent by the Byzantine attackers from noisy stochastic gradients sent by the ‘honest’ workers. This motivates reducing the variance of stochastic gradients as a means of robustifying SGD. To this end, a novel Byzantine attack resilient distributed (Byrd-) SAGA approach is introduced for federated learning tasks involving multiple workers. Rather than the mean employed by distributed SAGA, the novel Byrd-SAGA relies on the geometric median to aggregate the corrected stochastic gradients sent by the workers. When less than half of the workers are Byzantine attackers, Byrd-SAGA attains provably linear convergence to a neighborhood of the optimal solution, with the asymptotic learning error determined by the number of Byzantine workers. Numerical tests corroborate the robustness to various Byzantine attacks, as well as the merits of Byrd-SAGA over Byzantine attack resilient distributed SGD.