PAC-Bayesian transfer learning: theory and algorithms

pac -贝叶斯理论由David McAllester（1999）提出，是一种机器学习的统计方法。它提供了预测精度的保证，用泛化误差的概率上界表示。经典PAC-Bayesian结果的主要假设是，观察到的训练集包含由（固定和未知的）数据分布产生的独立和同分布（iid）观测值，并且学习到的预测器将用于由相同数据分布产生的观测值。这个假设并不适用于许多现实生活中的机器学习任务，因为数据分布容易随着时间和环境的变化而变化。为了避免为每个新的数据分布重新训练机器学习（这需要为新任务提供适当的训练集），已经开发了几种迁移学习方法。迁移学习框架包括所有通过解决第一个任务来解决第二个任务（不同但与第一个任务相关）的方法。在有利的情况下，只需要少量的训练观察就可以使预测器适应新的任务。注意，根据问题的特点，存在几种迁移学习方法：领域适应、多任务学习、元学习等。一个有趣的框架是表征学习框架，它在最近的深度神经网络进展中得到了推广。本研究计划侧重于利用PAC-Bayesian理论研究各种迁移学习方法，分为两个主要研究轴。第一个轴将是研究无监督表示学习框架到pac -贝叶斯范式。一个有前途的研究方向是Arora等人（2019）最近提出的“对比”表示学习的严格理论形式化。它提供了一个坚实的基础，可以从数学上研究手头的问题。结果分析将用于神经网络的学习表示，以及其他作为核的方法。第二个轴将是建立在Germain等人（2016）的领域自适应算法的基础上，称为DALC。尽管该算法在小数据集上是有效的，但由于算法的复杂性，该算法与其他经典核方法一样，不能用于大数据集。一种众所周知的加速核方法的方法是随机傅立叶特征的近似（Rahimi和Recht， 2007），我们从pac -贝叶斯的角度重新审视了它（Letarte等人，2019）。另一个策略将是开发pac -贝叶斯最优传输方法来进行域适应。

The PAC-Bayesian theory, initiated by David McAllester (1999), is a statistical approach to machine learning. It provides guarantees on predictor accuracy, expressed by probabilistic upper bounds on the generalization error. The main assumption of the classical PAC-Bayesian results is that the observed training set contains independently and identically distributed (iid) observations generated by a (fixed and unknown) data distribution, and that the learned predictor is to be used on observations generated by the very same data distribution. This assumption does not hold for many real-life machine learning tasks, as the data distribution is prone to evolve with time and environment. To avoid retraining a machine learning from scratch for every new data distribution - which requires a proper training set for the new task(s) - several transfer learning methods have been developed. The transfer learning framework encompasses all methods reusing the "knowledge" learned by solving a first task in order to solve a second task (different but related to the first one). In a favorable context, only few training observations might be necessary to adapt the predictor to the new task. Note that there exist several transfer learning approaches, according to the problem characteristics: domain adaptation, multitask learning, meta-learning, etc. A framework of interest is the representation learning one, which has been popularized by the recent deep neural networks advances. This research proposal focuses on leveraging the PAC-Bayesian theory to study a variety of transfer learning approaches, divided into two main research axes. A fist axis will be to study the unsupervised representation learning framework into the PAC-Bayesian paradigm. A promising research direction is the rigorous theoretical formalization of "contrastive" representation learning proposed recently by Arora et al. (2019). It gives a solid basis from which one can mathematically study the problem at hand. The resulting analysis will be used for learning representation with neural networks, but also other methods as kernel ones. A second axis will be to build on the domain adaptation algorithm of Germain et al. (2016), named DALC. Although efficient on small datasets, this algorithm, as other classical kernel methods, cannot be used on large datasets due to its algorithmic complexity. A well-known method to accelerate kernel methods is the approximation by random Fourier features (Rahimi and Recht, 2007), that we revisited from a PAC-Bayesian perspective (Letarte et al., 2019). Another strategy will be to develop a PAC-Bayesian optimal transport approach to domain adaptation.