The proximal Robbins–Monro method
The proximal Robbins–Monro method
复制标题
近端罗宾斯-门罗法
DOI：
10.1111/rssb.12405
复制
发表时间：
2015
期刊：
影响因子：
--
通讯作者：
E. Airoldi
中科院分区：
文献类型：
--
作者：
Panos Toulis;Thibaut Horel;E. Airoldi
对大数据集的统计估计的需求重新激发了人们对迭代过程和随机优化的兴趣。随机近似是在最近的发展，因为它们产生的程序是简单的，一般的和快速的最前沿。然而，标准的随机近似往往是数值不稳定的。相反，确定性优化越来越多地使用最近更新来以原则性的方式实现数值稳定性。因此出现了理论上的空白。虽然标准的随机近似被包含在Robbins和Monro的框架中（The annals of mathematical statistics，1951，pp. 400-407），对于具有邻近更新的随机近似没有这样的框架。在本文中，我们概念化的经典罗宾斯-门罗程序的近端版本。我们的理论分析表明，所提出的程序具有重要的稳定性优于经典的Robbins-Monro程序，同时它保留了最好的已知的收敛速度。近似Robbins-Monro过程的精确实现具有挑战性，但我们表明近似实现导致易于实现的过程，并且通过实现数值稳定性仍然主导标准过程，实际上没有权衡。此外，近似近似
The need for statistical estimation with large data sets has reinvigorated interest in iterative procedures and stochastic optimization. Stochastic approximations are at the forefront of this recent development as they yield procedures that are simple, general and fast. However, standard stochastic approximations are often numerically unstable. Deterministic optimization, in contrast, increasingly uses proximal updates to achieve numerical stability in a principled manner. A theoretical gap has thus emerged. While standard stochastic approximations are subsumed by the framework Robbins and Monro (The annals of mathematical statistics, 1951, pp. 400–407), there is no such framework for stochastic approximations with proximal updates. In this paper, we conceptualize a proximal version of the classical Robbins–Monro procedure. Our theoretical analysis demonstrates that the proposed procedure has important stability benefits over the classical Robbins–Monro procedure, while it retains the best known convergence rates. Exact implementations of the proximal Robbins–Monro procedure are challenging, but we show that approximate implementations lead to procedures that are easy to implement, and still dominate standard procedures by achieving numerical stability, practically without trade‐offs. Moreover, approximate proximal Robbins–Monro procedures can be applied even when the objective cannot be calculated analytically, and so they generalize stochastic proximal procedures currently in use.
DOI：
10.1073/pnas.1908018116
发表时间：
2019-11-12
影响因子：
11.1
作者：
Asi, Hilal;Duchi, John C.
通讯作者：
Duchi, John C.