The Risks of Invariant Risk Minimization
The Risks of Invariant Risk Minimization
复制标题
DOI：
--
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Elan Rosenfeld;Pradeep Ravikumar;Andrej Risteski
中科院分区：
文献类型：
--
作者：
Elan Rosenfeld;Pradeep Ravikumar;Andrej Risteski
不变因果预测（Peters等人，2016）是一种用于分布外泛化的技术，其假设数据分布的某些方面在训练集中有所不同，但潜在的因果机制保持不变。最近，Arjovsky等人（2019年）提出了不变风险最小化（Invariant Risk Minimization，简称Risk Minimization），这是一个基于这一思想的目标，用于学习数据的深度不变特征，这些特征是潜在变量的复杂函数;随后提出了许多替代方案。然而，对所有这些工程的正式保障严重缺乏。在本文中，我们提出了第一次分析的分类下的目标$-$以及这些最近提出的替代品$-$下一个相当自然和一般的模型。在线性的情况下，我们展示了简单的条件下，最佳解决方案成功，或更经常地，未能恢复最佳不变预测。此外，我们提出了在非线性制度的第一个结果：我们证明，除非测试数据是足够相似的训练分布$-$，这正是它的目的是解决的问题，否则，可失败的灾难。因此，在这种情况下，我们发现，在标准的经验风险最小化的基础上，风险最小化及其替代方案根本没有改善。
Invariant Causal Prediction (Peters et al., 2016) is a technique for out-of-distribution generalization which assumes that some aspects of the data distribution vary across the training set but that the underlying causal mechanisms remain constant. Recently, Arjovsky et al. (2019) proposed Invariant Risk Minimization (IRM), an objective based on this idea for learning deep, invariant features of data which are a complex function of latent variables; many alternatives have subsequently been suggested. However, formal guarantees for all of these works are severely lacking. In this paper, we present the first analysis of classification under the IRM objective$-$as well as these recently proposed alternatives$-$under a fairly natural and general model. In the linear case, we show simple conditions under which the optimal solution succeeds or, more often, fails to recover the optimal invariant predictor. We furthermore present the very first results in the non-linear regime: we demonstrate that IRM can fail catastrophically unless the test data are sufficiently similar to the training distribution$-$this is precisely the issue that it was intended to solve. Thus, in this setting we find that IRM and its alternatives fundamentally do not improve over standard Empirical Risk Minimization.