FEDNEST: Federated Bilevel, Minimax, and Compositional Optimization
FEDNEST: Federated Bilevel, Minimax, and Compositional Optimization
复制标题
DOI：
10.48550/arxiv.2205.02215
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Davoud Ataee Tarzanagh;Mingchen Li;Christos Thrampoulidis;Samet Oymak
中科院分区：
文献类型：
--
作者：
Davoud Ataee Tarzanagh;Mingchen Li;Christos Thrampoulidis;Samet Oymak
标准的联邦优化方法成功地应用于具有单层级结构的随机问题。然而，许多当代的机器学习问题——包括对抗鲁棒性、超参数调整以及演员 - 评论家算法——都属于嵌套的双层规划问题，其中包含了极小极大和复合优化。在这项工作中，我们提出了\(FedBLO\)：一种联邦交替随机梯度方法来解决一般的嵌套问题。我们在存在异构数据的情况下为\(FedBLO\)建立了可证明的收敛速度，并针对双层、极小极大和复合优化引入了变体。\(FedBLO\)引入了多项创新，包括联邦超梯度计算和方差减少，以解决内层异构性问题。我们通过超参数\(-\)表示学习和极小极大优化的实验对我们的理论进行了补充，这些实验证明了我们的方法在实践中的优势。代码可在https://github.com/ucr - optml/FedNest获取。
Standard federated optimization methods successfully apply to stochastic problems with single-level structure. However, many contemporary ML problems -- including adversarial robustness, hyperparameter tuning, and actor-critic -- fall under nested bilevel programming that subsumes minimax and compositional optimization. In this work, we propose \fedblo: A federated alternating stochastic gradient method to address general nested problems. We establish provable convergence rates for \fedblo in the presence of heterogeneous data and introduce variations for bilevel, minimax, and compositional optimization. \fedblo introduces multiple innovations including federated hypergradient computation and variance reduction to address inner-level heterogeneity. We complement our theory with experiments on hyperparameter \-representation learning and minimax optimization that demonstrate the benefits of our method in practice. Code is available at https://github.com/ucr-optml/FedNest.