On the Generalization Error of Stochastic Mirror Descent for Quadratically-Bounded Losses: an Improved Analysis
On the Generalization Error of Stochastic Mirror Descent for Quadratically-Bounded Losses: an Improved Analysis
复制标题
DOI：
--
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Ta Duy Nguyen;Alina Ene;Huy Nguyen
中科院分区：
文献类型：
--
作者：
Ta Duy Nguyen;Alina Ene;Huy Nguyen
在这项工作中，我们重新审视了Telgarsky（2022）研究的1次二次有界损失的随机镜像下降的推广误差。二次有界2损失是一类广泛的损失函数，捕获Lipschitz和光滑3函数，用于回归和分类问题。我们研究了高4概率推广这类损失的线性预测在5实现和不可实现的情况下，当数据采样IID或从6马尔可夫链。先前的工作依赖于一个复杂的耦合参数之间的原始问题的迭代和那些投影到一个有界域。[8]这种方法允许集中不等式的黑盒应用，但[9]也导致次优保证，部分原因是在所有迭代中使用了联合边界[10]。在这项工作中，我们显着偏离11 Telgarsky（2022）的先前工作，并介绍了一种新的方法来建立高概率12泛化保证。与以前的工作相比，我们的工作直接分析了一个新的上鞅序列的矩生成函数，并利用了随机镜像下降的结构。因此，我们在所有上述设置中获得改进的边界15。具体来说，在可实现的情况下和不可实现的情况下，16轻尾亚高斯数据，我们提高了一个log T因子的界限，17匹配的正确率1 /T和1 /T/T，分别。在更具挑战性的18例重尾多项式数据，我们改进了现有的限制聚T19因子。20
In this work, we revisit the generalization error of stochastic mirror descent for 1 quadratically bounded losses studied in Telgarsky (2022). Quadratically bounded 2 losses is a broad class of loss functions, capturing both Lipschitz and smooth 3 functions, for both regression and classification problems. We study the high 4 probability generalization for this class of losses on linear predictors in both 5 realizable and non-realizable cases when the data are sampled IID or from a 6 Markov chain. The prior work relies on an intricate coupling argument between 7 the iterates of the original problem and those projected onto a bounded domain. 8 This approach enables blackbox application of concentration inequalities, but 9 also leads to suboptimal guarantees due in part to the use of a union bound 10 across all iterations. In this work, we depart significantly from the prior work of 11 Telgarsky (2022), and introduce a novel approach for establishing high probability 12 generalization guarantees. In contrast to the prior work, our work directly analyzes 13 the moment generating function of a novel supermartingale sequence and leverages 14 the structure of stochastic mirror descent. As a result, we obtain improved bounds 15 in all aforementioned settings. Specifically, in the realizable case and non-realizable 16 case with light-tailed sub-Gaussian data, we improve the bounds by a log T factor, 17 matching the correct rates of 1 /T and 1 / √ T , respectively. In the more challenging 18 case of heavy-tailed polynomial data, we improve the existing bound by a poly T 19 factor. 20