MLWiNS: Wireless On-the-Edge Training of Deep Networks Using Independent Subnets

神经网络(NN)最近在机器学习(ML)方面取得了许多成功。然而，这种成功是以高昂的成本为代价的：为了获得更好的ML模型，需要训练和部署越来越大的NN。这对于移动ML应用来说是一个问题，其中模型训练和推理需要在计算/通信轻量级和能量受限的平台上以及时的方式进行。这类应用必须运行在手持设备或无人机和EDGE基础设施上，并带来了新的挑战：EDGE网络的异构性、移动设备的不可靠性、此类设备的计算和能源限制以及无线网络中的通信瓶颈。这个项目将通过研究一种新的计算和通信轻、能量受限的分布式神经网络学习范例来解决这些挑战。这个项目的成功将产生基本的想法和工具，使移动分布式学习变得实用。此外，该项目将产生能够吸引不同学生群体的课程和开放教育资源。所研究的具体思想是一类新的分布式神经网络训练算法，称为独立子网络训练(IST)。IST将一个NN分解成一组独立的子网络。在同步步骤之前，针对一个或多个反向传播步骤，在不同的设备处训练这些子网络中的每一个。在下一轮分解和本地训练之前，更新的子网络被从边缘设备发送到参数服务器，用于重新组装成原始NN。因为子网络不共享参数，所以同步不需要聚合--它只需要交换参数。此外，每个子网络本身都是一个完全可操作的分类器；不需要子网络之间的同步管道。提出的IST的主要优点是：i)IST每次迭代分配给每个移动节点的训练参数更少，显著减少了通信开销；ii)每个设备训练的模型要小得多，从而导致更少的计算成本和更好的能量消耗。因此，我们有充分的理由期待IST将比移动应用程序的经典训练算法扩展得更好。该项目将研究如何将IST整合/扩展到各种NN架构，开发解释IST效率的新理论，并通过提出针对移动应用的纵向和横向硬件级系统实现将理论与实践相结合。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Neural networks (NN) have led to many recent successes in machine learning (ML). However, this success comes at a prohibitive cost: to obtain better ML models, larger and larger NNs need to be trained and deployed. This is a problem for mobile ML applications, where model training and inference need to be carried out in a timely fashion on a computation-/communication-light, and energy-limited platform. Such applications must run on handheld devices or drones and edge infrastructure and introduce new challenges: the heterogeneity of edge networks, the unreliability of the mobile devices, the computational and energy restrictions on such devices, and the communication bottlenecks in wireless networks. This project will address these challenges by investigating a new paradigm for computation- and communication-light, energy-limited distributed NN learning. Success in this project will produce fundamental ideas and tools that will make mobile distributed learning practical. Further, the project will generate courses and open-education resources that can attract diverse groups of students. The specific idea investigated is a new class of distributed NN training algorithms, called independent subnetwork training (IST). IST decomposes a NN into a set of independent subnetworks. Each of those subnetworks is trained at a different device, for one or more backpropagation steps, before a synchronization step. Updated subnetworks are sent from edge-devices to the parameter server for reassembly into the original NN, before the next round of decomposition and local training. Because the subnetworks share no parameters, synchronization requires no aggregation—it is just an exchange of parameters. Moreover, each of the subnetworks is a fully operational classifier by itself; no synchronization pipelines between subnetworks are required. Key benefits of the proposed IST are that: i) IST assigns fewer training parameters to each mobile node per iteration, significantly reducing the communication overhead, and ii) each device trains a much smaller model, resulting in less computational costs and better energy consumption. Thus, there is good reason to expect that IST will scale much better than classic training algorithms for mobile applications. The project will investigate how to incorporate/extend IST to various NN architectures, develop new theories that explain the efficiency of IST, and unify theory with practice by proposing hardware-level system implementations that scale up and out for mobile applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.