Exploiting Feature Heterogeneity for Improved Generalization in Federated Multi-task Learning
Exploiting Feature Heterogeneity for Improved Generalization in Federated Multi-task Learning
复制标题
DOI：
10.1109/isit54713.2023.10206757
复制
发表时间：
2023-06
期刊：
影响因子：
--
通讯作者：
Renpu Liu;Jing Yang;Cong Shen
中科院分区：
文献类型：
--
作者：
Renpu Liu;Jing Yang;Cong Shen
在这项工作中，我们研究了一个一般的联邦多任务学习（FMTL）问题，其中每个任务可以在多个客户端执行，每个客户端可以执行多个任务。虽然任务共享一些共同的表示（即，特征图），可以帮助学习，特征空间中的特征分布可能会在不同客户端的不同任务中有所不同，这对FMTL构成了重大挑战。虽然不同客户端的非独立和相同分布（非IID）本地数据集通常被认为不利于联邦学习（FL）中的模型收敛，但特征空间中的这种统计异质性可能有利于泛化性能。在这项工作中，我们建立了统计特征异质性的影响，通过透镜的多任务线性回归模型的泛化。为了利用特征分布的异质性，我们提出了一种新的基于增强数据集的方法，并证明了在一定条件下，异构数据集上的FMTL可以在泛化性能方面优于同质对应物。通过理论分析，提出了一种基于优化超额风险上界的简单客户加权方法。实验结果表明，该方法可以提高泛化性能的真实世界的数据集。
In this work, we investigate a general federated multitask learning (FMTL) problem where each task may be performed at multiple clients, and each client may perform multiple tasks. Although the tasks share some common representation (i.e., feature-map) that can help to learn, the distribution of the features in the feature space may vary across different tasks at different clients, which poses a significant challenge to FMTL. While non-independent and identically distributed (non-IID) local datasets at different clients are often considered detrimental to model convergence in federated learning (FL), such statistical heterogeneity in feature space may be beneficial to the generalization performance. In this work, we establish the impact of statistical feature heterogeneity on generalization, through the lens of a multi-task linear regression model. In order to leverage the feature distribution heterogeneity, we propose a novel augmented dataset based approach, and prove that under certain conditions, FMTL on heterogeneous datasets can outperform the homogeneous counterpart in terms of the generalization performance. The theoretical analysis further leads to a simple client weighting method based on optimizing the excess risk upper bound. Experimental results demonstrate that the generalization performance can be improved on a real-world dataset with the proposed method.