Understanding Why Generalized Reweighting Does Not Improve Over ERM
Understanding Why Generalized Reweighting Does Not Improve Over ERM
复制标题
DOI：
--
复制
发表时间：
2022-01
期刊：
影响因子：
--
通讯作者：
Runtian Zhai;Chen Dan;Zico Kolter;Pradeep Ravikumar
中科院分区：
文献类型：
--
作者：
Runtian Zhai;Chen Dan;Zico Kolter;Pradeep Ravikumar
经验风险最小化（ERM）在实践中是已知的，是不鲁棒的分布转移的训练和测试分布是不同的。一套方法，如重要性加权，和变体的分布鲁棒优化（DRO），已被提出来解决这个问题。但最近的一系列工作经验表明，这些方法并没有显着改善在真实的应用程序与分布转移的ERM。这项工作的目标是获得一个全面的理论理解这一有趣的现象。我们首先将广义重加权（GRW）算法的类，作为一个广泛的类别的方法，迭代更新模型参数的训练样本的迭代重加权的基础上。我们表明，当过参数化模型在GRW下训练时，得到的模型接近ERM获得的模型。我们还表明，添加小的正则化不会对经验训练精度产生很大影响，但这并没有帮助。总之，我们的研究结果表明，一个广泛的类别，我们称之为GRW的方法是无法实现分布鲁棒的泛化。因此，我们的工作有以下发人深省的外卖：取得进展的分布鲁棒的推广，我们要么开发非GRW的方法，或者也许设计新的分类/回归损失函数，适应类的GRW方法。
Empirical risk minimization (ERM) is known in practice to be non-robust to distributional shift where the training and the test distributions are different. A suite of approaches, such as importance weighting, and variants of distributionally robust optimization (DRO), have been proposed to solve this problem. But a line of recent work has empirically shown that these approaches do not significantly improve over ERM in real applications with distribution shift. The goal of this work is to obtain a comprehensive theoretical understanding of this intriguing phenomenon. We first posit the class of Generalized Reweighting (GRW) algorithms, as a broad category of approaches that iteratively update model parameters based on iterative reweighting of the training samples. We show that when overparameterized models are trained under GRW, the resulting models are close to that obtained by ERM. We also show that adding small regularization which does not greatly affect the empirical training accuracy does not help. Together, our results show that a broad category of what we term GRW approaches are not able to achieve distributionally robust generalization. Our work thus has the following sobering takeaway: to make progress towards distributionally robust generalization, we either have to develop non-GRW approaches, or perhaps devise novel classification/regression loss functions that are adapted to the class of GRW approaches.