PAC-Bayesian transfer learning: theory and algorithms

由David McAllester(1999)提出的PAC-贝叶斯理论是机器学习的一种统计方法。它提供了预测器精度的保证，用泛化误差的概率上界表示。经典PAC-贝叶斯结果的主要假设是，观察到的训练集包含由(固定的和未知的)数据分布生成的独立且同分布(IID)的观测值，并且学习的预测器将用于由完全相同的数据分布生成的观测值。这一假设不适用于许多现实生活中的机器学习任务，因为数据分布容易随着时间和环境而演变。 为了避免为每个新的数据分布从头开始重新训练机器学习-这需要为新任务设置适当的训练集(S)-已经开发了几种迁移学习方法。迁移学习框架涵盖了通过解决第一个任务而学到的所有方法，以解决第二个任务(不同于第一个任务，但与第一个任务相关)。在有利的情况下，可能只需要很少的训练观察就可以使预测器适应新任务。注意，根据问题的特点，有几种迁移学习方法：领域自适应、多任务学习、元学习等。一种感兴趣的框架是表征学习框架，它已被最近的深度神经网络的进展所普及。本研究方案侧重于利用PAC-贝叶斯理论对多种迁移学习方法进行研究，分为两个主要研究轴线。 第一个轴心将是将无监督表示学习框架研究到PAC-贝叶斯范式中。Arora等人最近提出的“对比”表征学习的严格理论形式化是一个很有前途的研究方向。(2019年)。它为人们从数学上研究手头的问题提供了坚实的基础。所得到的分析结果将用于神经网络的学习表示，但也将用于其他方法作为核心方法。 第二个轴将建立在Germain等人的域适应算法的基础上。(2016)，命名为DALC。虽然该算法在小数据集上是有效的，但与其他经典的核方法一样，由于算法的复杂性，它不能用于大数据集。一种众所周知的加速核方法的方法是随机傅立叶特征逼近(Rahimi和Recht，2007)，我们从PAC-Bayes的角度重新讨论了这一点(Letarte等人，2019年)。另一种策略将是开发一种PAC-贝叶斯最优传输方法来实现域适应。

The PAC-Bayesian theory, initiated by David McAllester (1999), is a statistical approach to machine learning. It provides guarantees on predictor accuracy, expressed by probabilistic upper bounds on the generalization error. The main assumption of the classical PAC-Bayesian results is that the observed training set contains independently and identically distributed (iid) observations generated by a (fixed and unknown) data distribution, and that the learned predictor is to be used on observations generated by the very same data distribution. This assumption does not hold for many real-life machine learning tasks, as the data distribution is prone to evolve with time and environment. To avoid retraining a machine learning from scratch for every new data distribution - which requires a proper training set for the new task(s) - several transfer learning methods have been developed. The transfer learning framework encompasses all methods reusing the "knowledge" learned by solving a first task in order to solve a second task (different but related to the first one). In a favorable context, only few training observations might be necessary to adapt the predictor to the new task. Note that there exist several transfer learning approaches, according to the problem characteristics: domain adaptation, multitask learning, meta-learning, etc. A framework of interest is the representation learning one, which has been popularized by the recent deep neural networks advances. This research proposal focuses on leveraging the PAC-Bayesian theory to study a variety of transfer learning approaches, divided into two main research axes. A fist axis will be to study the unsupervised representation learning framework into the PAC-Bayesian paradigm. A promising research direction is the rigorous theoretical formalization of "contrastive" representation learning proposed recently by Arora et al. (2019). It gives a solid basis from which one can mathematically study the problem at hand. The resulting analysis will be used for learning representation with neural networks, but also other methods as kernel ones. A second axis will be to build on the domain adaptation algorithm of Germain et al. (2016), named DALC. Although efficient on small datasets, this algorithm, as other classical kernel methods, cannot be used on large datasets due to its algorithmic complexity. A well-known method to accelerate kernel methods is the approximation by random Fourier features (Rahimi and Recht, 2007), that we revisited from a PAC-Bayesian perspective (Letarte et al., 2019). Another strategy will be to develop a PAC-Bayesian optimal transport approach to domain adaptation.