Analysis of Trainability of Gradient-based Multi -environment Learning from Gradient Norm Regularization Perspective
Analysis of Trainability of Gradient-based Multi -environment Learning from Gradient Norm Regularization Perspective
复制标题
DOI：
10.1109/ijcnn52387.2021.9533904
复制
发表时间：
2021-07
期刊：
影响因子：
--
通讯作者：
S. Takagi;Yoshihiro Nagano;Yuki Yoshida;Masato Okada
中科院分区：
文献类型：
--
作者：
S. Takagi;Yoshihiro Nagano;Yuki Yoshida;Masato Okada
对多种环境的适应性和不变性都是智能系统的关键能力。模型不可知元学习(MAML)是一种能够实现这种适应性的元学习算法，不变风险最小化(IRM)是实现跨环境不变表示的问题设置。我们可以将这两种方法描述为具有环境相关约束的优化问题，而这种约束会阻碍优化。因此，了解约束对优化的影响非常重要。在本文中，我们通过分析具有梯度范数惩罚的损失的梯度下降的可训练性，对约束如何影响MAML和IRM的优化提供了一个概念性的见解，它更容易研究，但与MAML和IRM有关。我们用MAML和IRM的实际数据集和结构进行了数值实验，验证了梯度范数惩罚损失分析很好地捕捉到了MAML和IRM的约束与可训练性之间的经验关系。
Adaptation and invariance to multiple environments are both crucial abilities for intelligent systems. Model-agnostic meta-learning (MAML) is a meta-learning algorithm to enable such adaptability, and invariant risk minimization (IRM) is a problem setting to achieve the invariant representation across multiple environments. We can formulate both methods as optimization problems with the environment-dependent constraint and this constraint is known to hamper optimization. Therefore, understanding the effect of the constraint on the optimization is important. In this paper, we provide a conceptual insight on how the constraint affects the optimization of MAML and IRM by analyzing the trainability of the gradient descent on the loss with the gradient norm penalty, which is easier to study but is related to both MAML and IRM. We conduct numerical experiments with practical datasets and architectures for MAML and IRM and validate that the analysis of the gradient norm penalty loss captures well the empirical relationship between the constraint and the trainability of MAML and IRM.