MLWiNS: Wireless On-the-Edge Training of Deep Networks Using Independent Subnets

MLWiNS：使用独立子网的深度网络无线边缘训练

• 批准号: 2003137
• 负责人: Christopher Jermaine
• 金额: $ 30万
• 依托单位: William Marsh Rice University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-06-01 至 2023-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2003137&HistoricalAwards=false
• 关键词: MLWiNS; Wireless; Edge; Training; Deep

Neural networks (NN) have led to many recent successes in machine learning (ML). However, this success comes at a prohibitive cost: to obtain better ML models, larger and larger NNs need to be trained and deployed. This is a problem for mobile ML applications, where model training and inference need to be carried out in a timely fashion on a computation-/communication-light, and energy-limited platform. Such applications must run on handheld devices or drones and edge infrastructure and introduce new challenges: the heterogeneity of edge networks, the unreliability of the mobile devices, the computational and energy restrictions on such devices, and the communication bottlenecks in wireless networks. This project will address these challenges by investigating a new paradigm for computation- and communication-light, energy-limited distributed NN learning. Success in this project will produce fundamental ideas and tools that will make mobile distributed learning practical. Further, the project will generate courses and open-education resources that can attract diverse groups of students. The specific idea investigated is a new class of distributed NN training algorithms, called independent subnetwork training (IST). IST decomposes a NN into a set of independent subnetworks. Each of those subnetworks is trained at a different device, for one or more backpropagation steps, before a synchronization step. Updated subnetworks are sent from edge-devices to the parameter server for reassembly into the original NN, before the next round of decomposition and local training. Because the subnetworks share no parameters, synchronization requires no aggregation—it is just an exchange of parameters. Moreover, each of the subnetworks is a fully operational classifier by itself; no synchronization pipelines between subnetworks are required. Key benefits of the proposed IST are that: i) IST assigns fewer training parameters to each mobile node per iteration, significantly reducing the communication overhead, and ii) each device trains a much smaller model, resulting in less computational costs and better energy consumption. Thus, there is good reason to expect that IST will scale much better than classic training algorithms for mobile applications. The project will investigate how to incorporate/extend IST to various NN architectures, develop new theories that explain the efficiency of IST, and unify theory with practice by proposing hardware-level system implementations that scale up and out for mobile applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

神经网络（NN）最近在机器学习（ML）方面取得了许多成功。然而，这种成功的代价是高昂的：为了获得更好的ML模型，需要训练和部署越来越大的NN。这对于移动的ML应用来说是一个问题，其中模型训练和推理需要在计算/通信轻和能量有限的平台上及时进行。这些应用程序必须在手持设备或无人机和边缘基础设施上运行，并引入新的挑战：边缘网络的异构性，移动的设备的不可靠性，这些设备上的计算和能源限制以及无线网络中的通信瓶颈。该项目将通过研究一种新的计算和通信轻型，能量有限的分布式NN学习范式来解决这些挑战。这个项目的成功将产生使移动的分布式学习实用的基本思想和工具。此外，该项目还将提供课程和开放教育资源，吸引不同群体的学生。研究的具体思想是一类新的分布式NN训练算法，称为独立子网络训练（IST）。IST将NN分解为一组独立的子网络。在同步步骤之前，针对一个或多个反向传播步骤，在不同的设备处训练这些子网络中的每一个。更新后的子网络从边缘设备发送到参数服务器，以便在下一轮分解和局部训练之前重新组装成原始NN。因为子网之间没有共享参数，所以同步不需要聚合--它只是交换参数。此外，每个子网络本身就是一个完全可操作的分类器;子网络之间不需要同步管道。所提出的IST的主要优点是：i）IST每次迭代向每个移动的节点分配更少的训练参数，从而显著降低通信开销，以及ii）每个设备训练小得多的模型，从而导致更少的计算成本和更好的能耗。因此，有充分的理由期待IST将比用于移动的应用的经典训练算法更好地扩展。该项目将研究如何将IST整合/扩展到各种NN架构，开发解释IST效率的新理论，并通过提出可扩展和扩展到移动的应用的硬件级系统实现来统一理论与实践。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估而被认为值得支持。

项目成果

Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?

• DOI: 10.48550/arxiv.2203.08392
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Y. Fu;Shunyao Zhang;Shan-Hung Wu;Cheng Wan;Yingyan Lin

BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling

• DOI: 10.48550/arxiv.2203.10983
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Cheng Wan;Youjie Li;Ang Li;Namjae Kim;Yingyan Lin

LDP: Learnable Dynamic Precision for Efficient Deep Neural Network Training and Inference

• DOI: 10.48550/arxiv.2203.07713
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Zhongzhi Yu;Y. Fu;Shang Wu;Mengquan Li;Haoran You;Yingyan Lin

PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communication

• DOI: 10.48550/arxiv.2203.10428
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Cheng Wan;Youjie Li;Cameron R. Wolfe;Anastasios Kyrillidis;Namjae Kim;Yingyan Lin