RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets
RSA: Byzantine-Robust Stochastic Aggregation Methods for Distributed Learning from Heterogeneous Datasets
复制标题
DOI：
10.1609/aaai.v33i01.33011544
复制
发表时间：
2018-11
期刊：
影响因子：
--
通讯作者：
Xiang Wang;Dingxian Wang;Canran Xu;Xiangnan He;Yixin Cao;Tat-Seng Chua
中科院分区：
文献类型：
--
作者：
Xiang Wang;Dingxian Wang;Canran Xu;Xiangnan He;Yixin Cao;Tat-Seng Chua
在本文中，我们提出了一类强大的随机亚级别方法，用于在存在未知数的拜占庭工人的情况下从异质数据集分布式学习。在学习过程中，拜占庭工人可能会由于数据损坏，通信失败或恶意攻击，因此可能会向主人发送任意错误的消息，从而偏向学习的模型。提出方法的关键是与目标函数合并的正规化术语，以鲁棒化学习任务并减轻拜占庭式攻击的负面影响。最终的基于亚级别的算法称为byzantine-bust随机聚集方法，证明我们的首字母缩写RSA从此以后使用。与大多数现有算法相反，RSA不依赖于数据是独立的，并且对工人的分布相同（I.I.D.），因此适合更广泛的应用程序。从理论上讲，我们表明：i）RSA收敛到近乎最佳的解决方案，其学习错误取决于拜占庭工人的数量； ii）在拜占庭式攻击下，RSA的收敛速率与没有拜占庭攻击的随机梯度下降法相同。从数值上讲，与最先进的替代方案相比，对实际数据集的实验证实了RSA的竞争性能和复杂性的降低。
In this paper, we propose a class of robust stochastic subgradient methods for distributed learning from heterogeneous datasets at presence of an unknown number of Byzantine workers. The Byzantine workers, during the learning process, may send arbitrary incorrect messages to the master due to data corruptions, communication failures or malicious attacks, and consequently bias the learned model. The key to the proposed methods is a regularization term incorporated with the objective function so as to robustify the learning task and mitigate the negative effects of Byzantine attacks. The resultant subgradient-based algorithms are termed Byzantine-Robust Stochastic Aggregation methods, justifying our acronym RSA used henceforth. In contrast to most of the existing algorithms, RSA does not rely on the assumption that the data are independent and identically distributed (i.i.d.) on the workers, and hence fits for a wider class of applications. Theoretically, we show that: i) RSA converges to a near-optimal solution with the learning error dependent on the number of Byzantine workers; ii) the convergence rate of RSA under Byzantine attacks is the same as that of the stochastic gradient descent method, which is free of Byzantine attacks. Numerically, experiments on real dataset corroborate the competitive performance of RSA and a complexity reduction compared to the state-of-the-art alternatives.