An Adaptive Half-Space Projection Method for Stochastic Optimization Problems with Group Sparse Regularization
An Adaptive Half-Space Projection Method for Stochastic Optimization Problems with Group Sparse Regularization
复制标题
DOI：
--
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Yutong Dai;Tianyi Chen;Guanyi Wang;Daniel P. Robinson
中科院分区：
文献类型：
--
作者：
Yutong Dai;Tianyi Chen;Guanyi Wang;Daniel P. Robinson
群稀疏正则化优化问题在各种流行的下游应用中普遍存在，例如深度神经网络（dnn）的特征选择和压缩。然而，当这种正则化与随机损失函数结合使用时，文献中现有的方法并没有表现得特别好。特别是，如何设计一种计算效率高、收敛保证且能计算群稀疏解的算法是一个挑战。最近提出的半空间随机投影梯度（HSPG）方法在一定程度上解决了这些挑战。本文提出了一个显著增强的HSPG版本，我们称之为AdaHSPG+，它有两个显著的进步。首先，与HSPG相比，AdaHSPG+在更宽松的假设条件下具有更强的收敛结果。这种收敛性的改进是通过将方差减少技术与迭代预测解决方案支持度的新自适应策略相结合来实现的。其次，与HSPG相比，AdaHSPG+需要更少的参数调优，从而使其更加实用和用户友好。这一进步是通过设计自动和自适应的策略来选择每次迭代所采用的步骤类型和更新关键的超参数来实现的。本文提出的AdaHSPG+算法在凸和非凸基准问题上的数值有效性得到了验证。源代码可从https://github.com/tianyic/adahspg获得。
Optimization problems with group sparse regularization are ubiquitous in various popular downstream applications, such as feature selection and compression for Deep Neural Networks (DNNs). Nonetheless, the existing methods in the literature do not perform particularly well when such regularization is used in combination with a stochastic loss function. In particular, it is challenging to design a computationally efficient algorithm with a convergence guarantee and can compute group-sparse solutions. Recently, a half-space stochastic projected gradient ( HSPG ) method was proposed that partly addressed these challenges. This paper presents a substantially enhanced version of HSPG that we call AdaHSPG+ that makes two noticeable advances. First, AdaHSPG+ is shown to have a stronger convergence result under significantly looser assumptions than those required by HSPG . This improvement in convergence is achieved by integrating variance reduction techniques with a new adaptive strategy for iteratively predicting the support of a solution. Second, AdaHSPG+ requires significantly less parameter tuning compared to HSPG , thus making it more practical and user-friendly. This advance is achieved by designing automatic and adaptive strategies for choosing the type of step employed at each iteration and for updating key hyperparam-eters. The numerical effectiveness of our proposed AdaHSPG+ algorithm is demonstrated on both convex and non-convex benchmark problems. The source code is available at https://github.com/tianyic/adahspg .