PAC-Bayesian transfer learning: theory and algorithms

点击翻译按钮获取中文摘要

The PAC-Bayesian theory, initiated by David McAllester (1999), is a statistical approach to machine learning. It provides guarantees on predictor accuracy, expressed by probabilistic upper bounds on the generalization error. The main assumption of the classical PAC-Bayesian results is that the observed training set contains independently and identically distributed (iid) observations generated by a (fixed and unknown) data distribution, and that the learned predictor is to be used on observations generated by the very same data distribution. This assumption does not hold for many real-life machine learning tasks, as the data distribution is prone to evolve with time and environment. To avoid retraining a machine learning from scratch for every new data distribution - which requires a proper training set for the new task(s) - several transfer learning methods have been developed. The transfer learning framework encompasses all methods reusing the "knowledge" learned by solving a first task in order to solve a second task (different but related to the first one). In a favorable context, only few training observations might be necessary to adapt the predictor to the new task. Note that there exist several transfer learning approaches, according to the problem characteristics: domain adaptation, multitask learning, meta-learning, etc. A framework of interest is the representation learning one, which has been popularized by the recent deep neural networks advances. This research proposal focuses on leveraging the PAC-Bayesian theory to study a variety of transfer learning approaches, divided into two main research axes. A fist axis will be to study the unsupervised representation learning framework into the PAC-Bayesian paradigm. A promising research direction is the rigorous theoretical formalization of "contrastive" representation learning proposed recently by Arora et al. (2019). It gives a solid basis from which one can mathematically study the problem at hand. The resulting analysis will be used for learning representation with neural networks, but also other methods as kernel ones. A second axis will be to build on the domain adaptation algorithm of Germain et al. (2016), named DALC. Although efficient on small datasets, this algorithm, as other classical kernel methods, cannot be used on large datasets due to its algorithmic complexity. A well-known method to accelerate kernel methods is the approximation by random Fourier features (Rahimi and Recht, 2007), that we revisited from a PAC-Bayesian perspective (Letarte et al., 2019). Another strategy will be to develop a PAC-Bayesian optimal transport approach to domain adaptation.