MLWiNS: Wireless On-the-Edge Training of Deep Networks Using Independent Subnets

神经网络（NN）最近在机器学习（ML）领域取得了许多成功。然而，这种成功的代价是高昂的：为了获得更好的ML模型，需要训练和部署越来越大的神经网络。这是移动机器学习应用程序的一个问题，其中模型训练和推理需要在计算/通信轻且能量有限的平台上及时进行。这些应用程序必须在手持设备或无人机和边缘基础设施上运行，并带来新的挑战：边缘网络的异构性、移动设备的不可靠性、此类设备的计算和能量限制以及无线网络中的通信瓶颈。该项目将通过研究计算和通信的新范式来解决这些挑战-轻，能量有限的分布式神经网络学习。这个项目的成功将产生基本的想法和工具，使移动分布式学习成为现实。此外，该项目将产生课程和开放教育资源，可以吸引不同群体的学生。研究的具体思想是一类新的分布式神经网络训练算法，称为独立子网训练（IST）。IST将一个神经网络分解成一组独立的子网。在同步步骤之前，每个子网都在不同的设备上进行一个或多个反向传播步骤的训练。在进行下一轮分解和局部训练之前，更新的子网从边缘设备发送到参数服务器，重新组装到原始神经网络中。由于子网不共享任何参数，因此同步不需要聚合——它只是交换参数。此外，每个子网本身都是一个完全可操作的分类器；子网间不需要同步管道。所提出的IST的主要优点是：i) IST每次迭代为每个移动节点分配的训练参数更少，显著降低了通信开销；ii)每个设备训练的模型更小，从而减少了计算成本和更好的能耗。因此，有充分的理由期待IST将比移动应用程序的经典训练算法扩展得更好。该项目将研究如何将IST整合/扩展到各种神经网络架构中，开发解释IST效率的新理论，并通过提出针对移动应用扩展和扩展的硬件级系统实现，将理论与实践结合起来。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Neural networks (NN) have led to many recent successes in machine learning (ML). However, this success comes at a prohibitive cost: to obtain better ML models, larger and larger NNs need to be trained and deployed. This is a problem for mobile ML applications, where model training and inference need to be carried out in a timely fashion on a computation-/communication-light, and energy-limited platform. Such applications must run on handheld devices or drones and edge infrastructure and introduce new challenges: the heterogeneity of edge networks, the unreliability of the mobile devices, the computational and energy restrictions on such devices, and the communication bottlenecks in wireless networks. This project will address these challenges by investigating a new paradigm for computation- and communication-light, energy-limited distributed NN learning. Success in this project will produce fundamental ideas and tools that will make mobile distributed learning practical. Further, the project will generate courses and open-education resources that can attract diverse groups of students. The specific idea investigated is a new class of distributed NN training algorithms, called independent subnetwork training (IST). IST decomposes a NN into a set of independent subnetworks. Each of those subnetworks is trained at a different device, for one or more backpropagation steps, before a synchronization step. Updated subnetworks are sent from edge-devices to the parameter server for reassembly into the original NN, before the next round of decomposition and local training. Because the subnetworks share no parameters, synchronization requires no aggregation—it is just an exchange of parameters. Moreover, each of the subnetworks is a fully operational classifier by itself; no synchronization pipelines between subnetworks are required. Key benefits of the proposed IST are that: i) IST assigns fewer training parameters to each mobile node per iteration, significantly reducing the communication overhead, and ii) each device trains a much smaller model, resulting in less computational costs and better energy consumption. Thus, there is good reason to expect that IST will scale much better than classic training algorithms for mobile applications. The project will investigate how to incorporate/extend IST to various NN architectures, develop new theories that explain the efficiency of IST, and unify theory with practice by proposing hardware-level system implementations that scale up and out for mobile applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.