Consensus Control for Decentralized Deep Learning
Consensus Control for Decentralized Deep Learning
复制标题
DOI：
--
复制
发表时间：
2021-02
影响因子：
7.4
通讯作者：
Lingjing Kong;Tao Lin;Anastasia Koloskova;Martin Jaggi;Sebastian U. Stich
中科院分区：
文献类型：
--
作者：
Lingjing Kong;Tao Lin;Anastasia Koloskova;Martin Jaggi;Sebastian U. Stich
深度学习模型的去中心化训练可以在网络上进行设备上的学习，也可以有效地扩展到大型计算集群。早期工作中的实验表明，即使在数据中心设置中，分散训练也经常受到模型质量下降的影响：以分散方式训练的模型的训练和测试性能通常比以集中方式训练的模型差，并且这种性能下降受到诸如网络大小，通信拓扑和数据分区等参数的影响。我们确定了设备之间不断变化的共识距离作为解释集中和分散训练之间差距的关键参数。我们从理论上证明，当训练共识距离低于一个临界值时，分散训练的收敛速度与集中训练的收敛速度一样快。我们通过经验验证了泛化性能与共识距离之间的关系与这一理论观察是一致的。我们的经验见解允许有原则地设计更好的分散训练方案，以减轻性能下降。为此，我们提供实用的培训指南，并举例说明其在数据中心设置中的有效性，作为重要的第一步。
Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. Experiments in earlier works reveal that, even in a data-center setup, decentralized training often suffers from the degradation in the quality of the model: the training and test performance of models trained in a decentralized fashion is in general worse than that of models trained in a centralized fashion, and this performance drop is impacted by parameters such as network size, communication topology and data partitioning. We identify the changing consensus distance between devices as a key parameter to explain the gap between centralized and decentralized training. We show in theory that when the training consensus distance is lower than a critical quantity, decentralized training converges as fast as the centralized counterpart. We empirically validate that the relation between generalization performance and consensus distance is consistent with this theoretical observation. Our empirical insights allow the principled design of better decentralized training schemes that mitigate the performance drop. To this end, we provide practical training guidelines and exemplify its effectiveness on the data-center setup as the important first step.