FABA: An Algorithm for Fast Aggregation against Byzantine Attacks in Distributed Neural Networks
FABA: An Algorithm for Fast Aggregation against Byzantine Attacks in Distributed Neural Networks
复制标题
DOI：
10.24963/ijcai.2019/670
复制
发表时间：
2019-08
期刊：
影响因子：
--
通讯作者：
Qi Xia;Zeyi Tao;Zijiang Hao;Qun A. Li
中科院分区：
文献类型：
--
作者：
Qi Xia;Zeyi Tao;Zijiang Hao;Qun A. Li
很多时候，对于复杂的网络模型来说，在单机上训练大规模深度学习神经网络变得越来越困难。分布式训练提供了有效的解决方案，但参与的worker可能会发生拜占庭攻击。它们可能会受到损害或遭受硬件故障的影响。如果他们上传有毒梯度，训练将变得不稳定，甚至收敛到鞍点。在本文中，我们提出了 FABA，一种针对拜占庭攻击的快速聚合算法，该算法去除上传梯度中的异常值并获得接近真实梯度的梯度。我们展示了我们的算法的收敛性。实验表明，我们的算法可以实现与非拜占庭情况类似的性能，并且与之前的算法相比，效率更高。
Many times, training a large scale deep learning neural network on a single machine becomes more and more difficult for a complex network model. Distributed training provides an efficient solution, but Byzantine attacks may occur on participating workers. They may be compromised or suffer from hardware failures. If they upload poisonous gradients, the training will become unstable or even converge to a saddle point. In this paper, we propose FABA, a Fast Aggregation algorithm against Byzantine Attacks, which removes the outliers in the uploaded gradients and obtains gradients that are close to the true gradients. We show the convergence of our algorithm. The experiments demonstrate that our algorithm can achieve similar performance to non-Byzantine case and higher efficiency as compared to previous algorithms.