Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks
Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Seyed Mohammadreza Mousavi Kalan;Zalan Fabian;A. Avestimehr;M. Soltanolkotabi
中科院分区：
文献类型：
--
作者：
Seyed Mohammadreza Mousavi Kalan;Zalan Fabian;A. Avestimehr;M. Soltanolkotabi
迁移学习已经成为一种强大的技术，可以提高机器学习模型在新领域的性能，在这些领域中，标记的训练数据可能很少。在这种方法中，为源任务训练的模型（其中有大量标记的训练数据可用）被用作在相关目标任务上训练模型的起点，而目标任务只有很少的标记的训练数据。尽管迁移学习方法最近取得了经验性的成功，但迁移学习的好处和基本限制却知之甚少。在本文中，我们开发了一个统计极小极大框架，以描述线性和单隐层神经网络模型回归背景下迁移学习的基本限制。具体来说，我们推导出一个下限的目标泛化误差可实现的任何算法作为一个函数的标记源和目标数据的数量，以及适当的概念之间的相似性的源和目标任务。我们的下限为迁移学习的好处和局限性提供了新的见解。我们进一步证实了我们的理论发现与各种实验。
Transfer learning has emerged as a powerful technique for improving the performance of machine learning models on new domains where labeled training data may be scarce. In this approach a model trained for a source task, where plenty of labeled training data is available, is used as a starting point for training a model on a related target task with only few labeled training data. Despite recent empirical success of transfer learning approaches, the benefits and fundamental limits of transfer learning are poorly understood. In this paper we develop a statistical minimax framework to characterize the fundamental limits of transfer learning in the context of regression with linear and one-hidden layer neural network models. Specifically, we derive a lower-bound for the target generalization error achievable by any algorithm as a function of the number of labeled source and target data as well as appropriate notions of similarity between the source and target tasks. Our lower bound provides new insights into the benefits and limitations of transfer learning. We further corroborate our theoretical finding with various experiments.