On the Convergence of Distributed Stochastic Bilevel Optimization Algorithms over a Network
On the Convergence of Distributed Stochastic Bilevel Optimization Algorithms over a Network
复制标题
DOI：
--
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Hongchang Gao;Bin Gu;M. Thai
中科院分区：
文献类型：
--
作者：
Hongchang Gao;Bin Gu;M. Thai
双层优化已广泛应用于各种机器学习模型，近年来开发了许多随机双层优化算法。然而，现有的算法大多局限于单机设置，无法处理分布式数据。为了解决这一问题，在所有参与者组成一个网络并在该网络中进行点对点通信的设置下，我们基于梯度跟踪通信机制和两种不同的梯度估计器开发了两种新的分散随机双层优化算法。此外，我们用新的理论分析策略建立了非凸强凸问题的收敛速率。据我们所知，这是第一次实现这些理论结果的工作。最后，我们将算法应用到实际的机器学习模型中，实验结果证实了算法的有效性。
Bilevel optimization has been applied to a wide variety of machine learning models, and numerous stochastic bilevel optimization algorithms have been developed in recent years. However, most existing algorithms restrict their focus on the single-machine setting so that they are incapable of handling the distributed data. To address this issue, under the setting where all participants compose a network and perform peer-to-peer communication in this network, we developed two novel decentralized stochastic bilevel optimization algorithms based on the gradient tracking communication mechanism and two different gradient estimators. Additionally, we established their convergence rates for nonconvex-strongly-convex problems with novel theoretical analysis strategies. To our knowledge, this is the first work achieving these theoretical results. Finally, we applied our algorithms to practical machine learning models, and the experimental results confirmed the efficacy of our algorithms.