The Effects of Weight Quantization on Online Federated Learning for the IoT: A Case Study
The Effects of Weight Quantization on Online Federated Learning for the IoT: A Case Study
复制标题
DOI：
10.1109/access.2024.3349557
复制
发表时间：
2024
期刊：
影响因子：
3.9
通讯作者：
Nil Llisterri Giménez;JunKyu Lee;Felix Freitag;Hans Vandierendonck
中科院分区：
文献类型：
--
作者：
Nil Llisterri Giménez;JunKyu Lee;Felix Freitag;Hans Vandierendonck
在使用高端计算机器的联邦学习中，许多权重量化方法被探索以节省客户端和服务器之间的通信带宽。然而，由于TinyML设备的硬件资源和功耗限制，TinyML设备受到小批量、神经网络规模和通信方式的限制，缺乏对在线联邦学习的权重量化研究。我们将Tiny Online Federated Learning（TinyOFL）命名为在物联网（IoT）中使用TinyML设备进行在线联合学习。本文从准确性、稳定性、过拟合、通信效率、能量消耗和传输时间等方面对TinyOFL中权重量化的影响进行了全面分析，并提取了如何将权重量化应用于TinyOFL的实用指南。我们的分析得到了TinyOFL案例研究的支持，其中三个Arduino Portenta H7板运行联邦学习客户端进行关键字定位任务。我们的研究结果包括，在TinyOFL中，可以允许比没有FL的在线学习更积极的权重量化，而不会影响准确性，这要归功于TinyOFL的准批量训练属性。例如，使用7位权重实现了与32位浮点权重相同的精度，同时节省了4.6\times $的通信带宽。通过增加网络宽度的过拟合在TinyOFL中很少发生，但如果应用强权重量化则可能发生。实验还表明，TinyOFL应用程序有一个设计空间，可以通过补偿由于权重量化而导致的精度损失来增加神经网络的大小。
Many weight quantization approaches were explored to save the communication bandwidth between the clients and the server in federated learning using high-end computing machines. However, there is a lack of weight quantization research for online federated learning using TinyML devices which are restricted by the mini-batch size, the neural network size, and the communication method due to their severe hardware resource constraints and power budgets. We name Tiny Online Federated Learning (TinyOFL) for online federated learning using TinyML devices in the Internet of Things (IoT). This paper performs a comprehensive analysis of the effects of weight quantization in TinyOFL in terms of accuracy, stability, overfitting, communication efficiency, energy consumption, and delivery time, and extracts practical guidelines on how to apply the weight quantization to TinyOFL. Our analysis is supported by a TinyOFL case study with three Arduino Portenta H7 boards running federated learning clients for a keyword spotting task. Our findings include that in TinyOFL, a more aggressive weight quantization can be allowed than in online learning without FL, without affecting the accuracy thanks to TinyOFL’s quasi-batch training property. For example, using 7-bit weights achieved the equivalent accuracy to 32-bit floating point weights, while saving communication bandwidth by $4.6 \times $ . Overfitting by increasing network width rarely occurs in TinyOFL, but may occur if strong weight quantization is applied. The experiments also showed that there is a design space for TinyOFL applications by compensating for the accuracy loss due to weight quantization with an increase of the neural network size.