Deep Kernel Methods

点击翻译按钮获取中文摘要

Deep neural networks are one of the most powerful modern machine learning methods, and can be applied to almost any domain where machine learning is useful, from understanding healthcare data to autonomous and adaptive robotic systems, to smart supply chains, video game design and content creation. The success of modern deep neural networks (DNNs) is based on their ability to use depth to transform the input into high-level representations that are good for solving difficult tasks. This is particularly evident in settings such as transfer learning, where a model is trained on a large scale dataset such as ImageNet, then the top-layer representation is used to solve a related task. However, the manner in which deep networks learn representations across layers is still poorly understood. In particular, perhaps the key theoretical approach for understanding DNN representations is to take an infinite-width limit. These infinite width limits highlight that representations can be formally described using kernels, which are matrices describing the network's judgement about the similarity of all pairs of input examples. However, the issue with infinite-width limits is that they eliminate representation learning; instead kernels become determinstic functions of the inputs, and are not learned based on the targets. Yang et al. (2021) recently introduced a new type of infinite-width limit that also retains representation learning, called the Deep Kernel Machine. It is the first entirely kernel-based deep learning method that gives comparable flexibility to DNNs. Kernel-based methods operate based on the similarity of different data points, which is fundamentally different from the feature-based regime utilised by neural networks, where learning is directly based on the value of data points. This new development shows promise, but the original paper only considers an analogue to simple fully-connected neural networks. Many of the interesting application domains utilise more complicated neural network architectures, such as convolutional neural networks for image tasks, or transformer networks and recurrent networks for sequential tasks like natural language processing. The goal of this project is to develop new extensions to the deep kernel machine literature (and perhaps more generally the deep kernel methods literature) to create deep kernel machine analogues to these complicated architectures, e.g. a convolutional deep kernel machine. Doing so will provide both a better theoretical understanding of how neural networks learn, which is very important for explainable AI, as well as new practical algorithms for performing supervised learning tasks. References: Yang, A., Robeyns, M., Milsom, E., Schoots, N., & Aitchison, L.. (2021). A theory of representation learning in deep neural networks gives a deep generalisation of kernel methods. https://www.ukri.org/what-we-offer/browse-our-areas-of-investment-and-support/artificial-intelligence-technologies/