TaskNorm: Rethinking Batch Normalization for Meta-Learning
TaskNorm: Rethinking Batch Normalization for Meta-Learning
复制标题
DOI：
--
复制
发表时间：
2020-03
期刊：
影响因子：
--
通讯作者：
J. Bronskill;Jonathan Gordon;James Requeima;Sebastian Nowozin;Richard E. Turner
中科院分区：
文献类型：
--
作者：
J. Bronskill;Jonathan Gordon;James Requeima;Sebastian Nowozin;Richard E. Turner
用于图像分类的现代元学习方法依赖于越来越深的网络来实现最先进的性能，使批量归一化成为元学习管道的重要组成部分。然而，元学习设置的分层性质提出了几个挑战，这些挑战可能会使传统的批量规范化无效，从而需要重新考虑这种设置中的规范化。我们评估了一系列用于元学习场景的批量标准化方法，并开发了一种称为TaskNorm的新方法。在14个数据集上的实验表明，批量归一化的选择对基于梯度和无梯度的元学习方法的分类精度和训练时间都有显着的影响。重要的是，TaskNorm被发现可以持续提高性能。最后，我们提供了一组规范化的最佳实践，可以对元学习算法进行公平的比较。
Modern meta-learning approaches for image classification rely on increasingly deep networks to achieve state-of-the-art performance, making batch normalization an essential component of meta-learning pipelines. However, the hierarchical nature of the meta-learning setting presents several challenges that can render conventional batch normalization ineffective, giving rise to the need to rethink normalization in this setting. We evaluate a range of approaches to batch normalization for meta-learning scenarios, and develop a novel approach that we call TaskNorm. Experiments on fourteen datasets demonstrate that the choice of batch normalization has a dramatic effect on both classification accuracy and training time for both gradient based and gradient-free meta-learning approaches. Importantly, TaskNorm is found to consistently improve performance. Finally, we provide a set of best practices for normalization that will allow fair comparison of meta-learning algorithms.