Dynamic Model Reduction for Decentralized Learning in Heterogeneous Mobile Networks
Dynamic Model Reduction for Decentralized Learning in Heterogeneous Mobile Networks
复制标题
DOI：
10.1109/mass58611.2023.00033
复制
发表时间：
2023-09
期刊：
影响因子：
--
通讯作者：
Sangsu Lee;Xi Zheng;Jie Hua;Haoxiang Yu;Christine Julien
中科院分区：
文献类型：
--
作者：
Sangsu Lee;Xi Zheng;Jie Hua;Haoxiang Yu;Christine Julien
与附近的设备合作，训练和个性化深度学习模型，为支持新的移动应用场景打开了潜力。在新兴的分散学习算法中，设备通过点对点网络进行通信，以共享从本地数据获得的知识。但是，通信带宽、计算能力和这些连接可用的持续时间是有限的，而且是异构的。在本文中，我们探讨了自适应模型约简策略用于分散学习算法（动态约简（DR））的可行性和有效性。在本研究中，我们使用现有的机会学习算法（OppCL）作为范例，该算法依赖于设备到设备的模型交换来迭代地训练基于遭遇的局部模型。在OppCL的分层模型约简中，当设备遇到潜在的学习伙伴时，通过量化权值，动态构建一个适合给定计算量和通信预算的模型约简，并构建一个dropout版本的神经网络。我们将我们的新方法命名为DR-OppCL，并表明DR-OppCL使用模拟和现实世界的迁移轨迹，以最小的努力调整与模型缩减相关的超参数，从而实现更快的收敛。虽然我们在OppCL的背景下展示了我们的DR方法，但它们是通用的，可以很容易地应用于其他分散学习算法。
Collaborating with nearby devices to train and personalize deep learning models opens the potential to support new mobile application scenarios. In emerging decentralized learning algorithms, devices communicate over a peer-to-peer network to share knowledge obtained from local data. However, communication bandwidth, computing power, and the duration for which these connections are available are limited and heterogeneous. In this paper, we explore the feasibility and efficacy of adaptive model reduction strategies for decentralized learning algorithms (Dynamic Reduction (DR)). For this study, we use as an exemplar an existing opportunistic learning algorithm (OppCL) that relies on device-to-device model exchanges to iteratively train a local model based on encounters. In layering model reduction on OppCL, when a device encounters a potential learning partner, it dynamically constructs a model reduction suitable for given computation and communication budget by quantizing weights and building a dropout version of a neural network. We term our new approach DR-OppCL and show that DR-OppCL leads to faster convergence with minimal effort in tuning hyperparameters related to model reduction, using both simulated and real-world mobility traces. While we demonstrate our DR approaches in the context of OppCL, they are generic and can be easily applied to other decentralized learning algorithms.