PAC-Bayesian transfer learning: theory and algorithms

PAC-Bayesian理论由大卫·麦卡莱斯特（David McAllester，1999）提出，是一种机器学习的统计方法。它提供了保证的预测精度，表示的概率上界的泛化误差。经典PAC-Bayesian结果的主要假设是，观察到的训练集包含由（固定和未知）数据分布生成的独立同分布（iid）观测，并且学习的预测器将用于由相同数据分布生成的观测。这个假设并不适用于许多现实生活中的机器学习任务，因为数据分布往往会随着时间和环境的变化而变化。 为了避免对每个新的数据分布从头开始重新训练机器学习-这需要为新任务提供适当的训练集-已经开发了几种迁移学习方法。迁移学习框架包含了所有的方法，这些方法重复使用通过解决第一个任务而学到的“知识”，以解决第二个任务（与第一个任务不同但相关）。在有利的情况下，可能只需要很少的训练观察就可以使预测器适应新的任务。请注意，根据问题的特征，存在几种迁移学习方法：域自适应，多任务学习，元学习等。感兴趣的框架是表示学习，它已被最近的深度神经网络进展所推广。该研究计划的重点是利用PAC-贝叶斯理论来研究各种迁移学习方法，分为两个主要研究轴。 第一个轴将研究无监督表示学习框架到PAC贝叶斯范式。一个有前途的研究方向是Arora等人最近提出的“对比”表征学习的严格理论形式化。它提供了一个坚实的基础，从中可以数学研究手头的问题。由此产生的分析将用于学习表示与神经网络，但也作为内核的其他方法。 第二个轴将建立在Germain等人的域自适应算法上。（2016），名为DALC。虽然该算法在小数据集上是有效的，但由于其算法复杂性，该算法与其他经典核方法一样不能用于大数据集。加速核方法的一种众所周知的方法是通过随机傅立叶特征的近似（Rahimi和Recht，2007），我们从PAC-贝叶斯的角度重新审视了该方法（Letarte等人，2019年）。另一个战略将是开发一个PAC贝叶斯最优传输方法域适应。

The PAC-Bayesian theory, initiated by David McAllester (1999), is a statistical approach to machine learning. It provides guarantees on predictor accuracy, expressed by probabilistic upper bounds on the generalization error. The main assumption of the classical PAC-Bayesian results is that the observed training set contains independently and identically distributed (iid) observations generated by a (fixed and unknown) data distribution, and that the learned predictor is to be used on observations generated by the very same data distribution. This assumption does not hold for many real-life machine learning tasks, as the data distribution is prone to evolve with time and environment. To avoid retraining a machine learning from scratch for every new data distribution - which requires a proper training set for the new task(s) - several transfer learning methods have been developed. The transfer learning framework encompasses all methods reusing the "knowledge" learned by solving a first task in order to solve a second task (different but related to the first one). In a favorable context, only few training observations might be necessary to adapt the predictor to the new task. Note that there exist several transfer learning approaches, according to the problem characteristics: domain adaptation, multitask learning, meta-learning, etc. A framework of interest is the representation learning one, which has been popularized by the recent deep neural networks advances. This research proposal focuses on leveraging the PAC-Bayesian theory to study a variety of transfer learning approaches, divided into two main research axes. A fist axis will be to study the unsupervised representation learning framework into the PAC-Bayesian paradigm. A promising research direction is the rigorous theoretical formalization of "contrastive" representation learning proposed recently by Arora et al. (2019). It gives a solid basis from which one can mathematically study the problem at hand. The resulting analysis will be used for learning representation with neural networks, but also other methods as kernel ones. A second axis will be to build on the domain adaptation algorithm of Germain et al. (2016), named DALC. Although efficient on small datasets, this algorithm, as other classical kernel methods, cannot be used on large datasets due to its algorithmic complexity. A well-known method to accelerate kernel methods is the approximation by random Fourier features (Rahimi and Recht, 2007), that we revisited from a PAC-Bayesian perspective (Letarte et al., 2019). Another strategy will be to develop a PAC-Bayesian optimal transport approach to domain adaptation.