FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Analysis
FL-NTK: A Neural Tangent Kernel-based Framework for Federated Learning Analysis
复制标题
DOI：
--
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Baihe Huang;Xiaoxiao Li;Zhao Song;Xin Yang
中科院分区：
文献类型：
--
作者：
Baihe Huang;Xiaoxiao Li;Zhao Song;Xin Yang
联邦学习（FL）是一种新兴的学习方案，允许不同的分布式客户端一起训练深度神经网络，而无需共享数据。神经网络因其前所未有的成功而变得流行。据我们所知，FL 关于具有显式形式和多步更新的神经网络的理论保证尚未被探索。尽管如此，FL 中神经网络的训练分析并不简单，原因有两个：首先，我们正在优化的目标损失函数是非平滑且非凸的，其次，我们甚至没有在梯度方向上进行更新。基于梯度下降的方法的现有收敛结果严重依赖于使用梯度方向进行更新的事实。本文提出了一种新的 FL 收敛分析，联邦学习神经正切核 (FL-NTK)，它对应于 FL 中通过梯度下降训练的过参数化 ReLU 神经网络，并受到神经正切核 (NTK) 分析的启发。理论上，FL-NTK 通过适当调整学习参数以线性速率收敛到全局最优解。此外，通过适当的分布假设，FL-NTK 还可以实现良好的泛化。
Federated Learning (FL) is an emerging learning scheme that allows different distributed clients to train deep neural networks together without data sharing. Neural networks have become popular due to their unprecedented success. To the best of our knowledge, the theoretical guarantees of FL concerning neural networks with explicit forms and multi-step updates are unexplored. Neverthe-less, training analysis of neural networks in FL is non-trivial for two reasons: first, the objective loss function we are optimizing is non-smooth and non-convex, and second, we are even not updating in the gradient direction. Existing convergence results for gradient descent-based methods heavily rely on the fact that the gradient direction is used for updating. This paper presents a new class of convergence analysis for FL, Federated Learning Neural Tangent Kernel (FL-NTK), which corresponds to overparamterized ReLU neural networks trained by gradient descent in FL and is inspired by the analysis in Neural Tangent Kernel (NTK). Theoretically, FL-NTK converges to a global-optimal solution at a linear rate with properly tuned learning parameters. Furthermore, with proper distributional assumptions, FL-NTK can also achieve good generalization.