Analysis of nonsmooth stochastic approximation: the differential inclusion approach
Analysis of nonsmooth stochastic approximation: the differential inclusion approach
复制标题
非光滑随机近似分析：微分包含法
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
É. Moulines
中科院分区：
文献类型：
--
作者：
Szymon Majewski;B. Miasojedow;É. Moulines
本文讨论了当极小化函数不是凸非光滑函数时，随机逼近的收敛问题。我们证明，对于光滑问题，收敛的“均值-极限”方法可以适用于非光滑情况。在适当的假设下，极限动力系统可以证明是一个微分包含。我们的结果扩展了Benaime等人在这个方向上的早期工作。(2005)，并提供了证明无约束和约束随机逼近问题的收敛的一般框架，其中显式或隐式更新。特别地，我们的结果允许我们建立在具有稀疏诱导惩罚的一大类深度学习和高维统计推理中产生的随机次梯度和近似式随机梯度下降算法的收敛。
In this paper we address the convergence of stochastic approximation when the functions to be minimized are not convex and nonsmooth. We show that the "mean-limit" approach to the convergence which leads, for smooth problems, to the ODE approach can be adapted to the non-smooth case. The limiting dynamical system may be shown to be, under appropriate assumption, a differential inclusion. Our results expand earlier works in this direction by Benaim et al. (2005) and provide a general framework for proving convergence for unconstrained and constrained stochastic approximation problems, with either explicit or implicit updates. In particular, our results allow us to establish the convergence of stochastic subgradient and proximal stochastic gradient descent algorithms arising in a large class of deep learning and high-dimensional statistical inference with sparsity inducing penalties.
影响因子：
3
作者：
Davis, Damek;Drusvyatskiy, Dmitriy;Lee, Jason D.
通讯作者：
Lee, Jason D.