Cross-Gradient Aggregation for Decentralized Learning from Non-IID data
Cross-Gradient Aggregation for Decentralized Learning from Non-IID data
复制标题
DOI：
--
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Yasaman Esfandiari;Sin Yong Tan;Zhanhong Jiang;Aditya Balu;Ethan Herron;C. Hegde;S. Sarkar
中科院分区：
文献类型：
--
作者：
Yasaman Esfandiari;Sin Yong Tan;Zhanhong Jiang;Aditya Balu;Ethan Herron;C. Hegde;S. Sarkar
分散式学习使一组协作代理能够使用分布式数据集学习模型，而无需中央参数服务器。最近，分散式学习算法在基准数据集上展示了最先进的结果，与集中式算法相当。然而，实现有竞争力的性能的关键假设是，数据是独立和相同的分布（IID）之间的代理，在现实生活中的应用程序，往往是不适用的。受持续学习思想的启发，我们提出了交叉梯度聚合（CGA），这是一种新的分散式学习算法，其中（i）每个代理聚合交叉梯度信息，即，其模型相对于其邻居的数据集的导数，以及（ii）使用基于二次规划（QP）的投影梯度来更新其模型。我们从理论上分析了CGA的收敛特性，并证明了它的效率非IID数据分布从MNIST和CIFAR-10数据集采样。我们的实证比较表明，上级学习性能的CGA比现有的国家的最先进的分散学习算法，以及保持信息压缩下的改进性能，以减少对等通信开销。代码可以在GitHub上找到。
Decentralized learning enables a group of collaborative agents to learn models using a distributed dataset without the need for a central parameter server. Recently, decentralized learning algorithms have demonstrated state-of-the-art results on benchmark data sets, comparable with centralized algorithms. However, the key assumption to achieve competitive performance is that the data is independently and identically distributed (IID) among the agents which, in real-life applications, is often not applicable. Inspired by ideas from continual learning, we propose Cross-Gradient Aggregation (CGA), a novel decentralized learning algorithm where (i) each agent aggregates cross-gradient information, i.e., derivatives of its model with respect to its neighbors' datasets, and (ii) updates its model using a projected gradient based on quadratic programming (QP). We theoretically analyze the convergence characteristics of CGA and demonstrate its efficiency on non-IID data distributions sampled from the MNIST and CIFAR-10 datasets. Our empirical comparisons show superior learning performance of CGA over existing state-of-the-art decentralized learning algorithms, as well as maintaining the improved performance under information compression to reduce peer-to-peer communication overhead. The code is available here on GitHub.