Network-Density-Controlled Decentralized Parallel Stochastic Gradient Descent in Wireless Systems
Network-Density-Controlled Decentralized Parallel Stochastic Gradient Descent in Wireless Systems
复制标题
DOI：
10.1109/icc40277.2020.9149125
复制
发表时间：
2020-02
期刊：
影响因子：
--
通讯作者：
Koya Sato;Yasuyuki Satoh;D. Sugimura
中科院分区：
文献类型：
--
作者：
Koya Sato;Yasuyuki Satoh;D. Sugimura
提出了一种用于无线系统分散学习的通信策略。我们的讨论是基于分散并行随机梯度下降（D-PSGD），这是最先进的分散学习算法之一。本文的主要贡献是为无线系统上的分散学习提出了一个新的开放问题：网络拓扑的密度可能会显著影响DPSGD的运行时性能。在实际的无线网络系统中，由于路径损耗和多径衰落，通常很难保证无延迟通信而不出现任何通信劣化。这些因素显著降低了D-PSGD的运行时性能。为了缓解这些问题，我们首先结合实际无线系统分析了D-PSGD的运行时性能。该分析得出了一些关键的见解，即密集网络拓扑(1)与稀疏网络拓扑相比，并没有显著提高D-PSGD的训练精度，并且(2)严重降低了运行时性能，因为这种设置通常需要利用低速率传输。基于这些发现，我们提出了一种新的通信策略，其中每个节点估计最优传输速率，使D-PSGD优化过程中的通信时间在网络密度的约束下最小化，网络密度以无线电传播特性为特征。该策略能够提高D-PSGD在无线系统中的运行性能。数值仿真结果表明，该策略能够提高D-PSGD的运行时性能。
This paper proposes a communication strategy for decentralized learning on wireless systems. Our discussion is based on the decentralized parallel stochastic gradient descent (D-PSGD), which is one of the state-of-the-art algorithms for decentralized learning. The main contribution of this paper is to raise a novel open question for decentralized learning on wireless systems: there is a possibility that the density of a network topology significantly influences the runtime performance of DPSGD. In general, it is difficult to guarantee delay-free communications without any communication deterioration in real wireless network systems because of path loss and multi-path fading. These factors significantly degrade the runtime performance of D-PSGD. To alleviate such problems, we first analyze the runtime performance of D-PSGD by considering real wireless systems. This analysis yields the key insights that dense network topology (1) does not significantly gain the training accuracy of D-PSGD compared to sparse one, and (2) strongly degrades the runtime performance because this setting generally requires to utilize a low-rate transmission. Based on these findings, we propose a novel communication strategy, in which each node estimates optimal transmission rates such that communication time during the D-PSGD optimization is minimized under the constraint of network density, which is characterized by radio propagation property. The proposed strategy enables to improve the runtime performance of D-PSGD in wireless systems. Numerical simulations reveal that the proposed strategy is capable of enhancing the runtime performance of D-PSGD.