Transfer Learning using Kolmogorov Complexity: Basic Theory and Empirical Evaluations
Transfer Learning using Kolmogorov Complexity: Basic Theory and Empirical Evaluations
复制标题
DOI：
--
复制
发表时间：
2007-12
期刊：
影响因子：
--
通讯作者：
M. H. Mahmud;S. Ray
中科院分区：
文献类型：
--
作者：
M. H. Mahmud;S. Ray
在迁移学习中，我们的目标是利用从解决相关问题中获得的信息，用更少的例子来解决新问题。迁移学习在实践中取得了成功，并对这些方法进行了广泛的PAC分析。然而，目前尚不清楚如何定义任务之间的相关性。这被认为是一个主要问题，因为它在概念上令人不安，它使人们不清楚要传输多少信息、何时以及如何传输信息。在本文中，我们提出使用任务之间的条件Kolmogorov复杂度来度量一个任务包含关于另一个任务的信息量。我们展示了现有的理论如何巧妙地解决了在贝叶斯设置下的顺序迁移学习中测量相关性和传递“正确”信息量的问题。该理论还表明，在非常正式和精确的意义上，没有其他合理的转移方法可以比我们的Kolmogorov复杂性理论转移方法做得更好，并且顺序转移总是合理的。我们还开发了一种实用的近似方法，并使用它在UCI ML存储库中任意选择的8个数据库之间传输信息。
In transfer learning we aim to solve new problems using fewer examples using information gained from solving related problems. Transfer learning has been successful in practice, and extensive PAC analysis of these methods has been developed. However it is not yet clear how to define relatedness between tasks. This is considered as a major problem as it is conceptually troubling and it makes it unclear how much information to transfer and when and how to transfer it. In this paper we propose to measure the amount of information one task contains about another using conditional Kolmogorov complexity between the tasks. We show how existing theory neatly solves the problem of measuring relatedness and transferring the 'right' amount of information in sequential transfer learning in a Bayesian setting. The theory also suggests that, in a very formal and precise sense, no other reasonable transfer method can do much better than our Kolmogorov Complexity theoretic transfer method, and that sequential transfer is always justified. We also develop a practical approximation to the method and use it to transfer information between 8 arbitrarily chosen databases from the UCI ML repository.