Dual Averaging Methods for Regularized Stochastic Learning and Online Optimization
Dual Averaging Methods for Regularized Stochastic Learning and Online Optimization
复制标题
DOI：
10.5555/1756006.1953017
复制
发表时间：
2009-12
期刊：
影响因子：
--
通讯作者：
Lin Xiao
中科院分区：
文献类型：
--
作者：
Lin Xiao
我们考虑正则化随机学习和在线优化问题，其中目标函数是两个凸项的和：一个是学习任务的损失函数，另一个是一个简单的正则化项，如l1-范数，用于提高稀疏性。我们开发了一种新的在线算法，正则化的双重平均（RDA）的方法，可以明确地利用在线设置的正则化结构。特别地，在每次迭代中，通过求解一个简单的优化问题来调整学习变量，该问题涉及损失函数和整个正则化项的所有过去子梯度的运行平均值，而不仅仅是其子梯度。计算实验表明，RDA方法可以非常有效的稀疏在线学习与l1-正则化。
We consider regularized stochastic learning and online optimization problems, where the objective function is the sum of two convex terms: one is the loss function of the learning task, and the other is a simple regularization term such as l1-norm for promoting sparsity. We develop a new online algorithm, the regularized dual averaging (RDA) method, that can explicitly exploit the regularization structure in an online setting. In particular, at each iteration, the learning variables are adjusted by solving a simple optimization problem that involves the running average of all past subgradients of the loss functions and the whole regularization term, not just its subgradient. Computational experiments show that the RDA method can be very effective for sparse online learning with l1-regularization.