Deep Kernel Methods

深度神经网络是最强大的现代机器学习方法之一，可以应用于几乎任何机器学习有用的领域，从理解医疗数据到自主和自适应机器人系统，再到智能供应链、视频游戏设计和内容创作。现代深度神经网络(DNN)的成功是基于它们使用深度将输入转换为有利于解决困难任务的高级表示的能力。这在转移学习等设置中尤其明显，在这种设置中，模型在大规模数据集(如ImageNet)上进行训练，然后使用顶层表示来解决相关任务。然而，深层网络学习跨层表示的方式仍然知之甚少。特别是，也许理解DNN表示的关键理论方法是取无限宽的限制。这些无限的宽度限制突出了表示可以使用核来形式化地描述，核是描述网络关于所有输入示例对的相似性的判断的矩阵。然而，无限宽度限制的问题是它们消除了表示学习；相反，核成为输入的确定性函数，而不是基于目标学习。Yang等人。(2021)最近引入了一种新的无限宽度限制，它也保留了表示学习，称为深度核机器。这是第一种完全基于核的深度学习方法，可以提供与DNN类似的灵活性。基于核的方法基于不同数据点的相似性，这与神经网络使用的基于特征的机制有根本不同，在神经网络中，学习直接基于数据点的价值。这一新的发展显示了希望，但最初的论文只考虑了一个简单的完全连接的神经网络的类比。许多有趣的应用领域使用更复杂的神经网络体系结构，例如用于图像任务的卷积神经网络，或用于自然语言处理等顺序任务的变压器网络和递归网络。该项目的目标是开发对深层内核机器文献(也许更一般的深层内核方法文献)的新扩展，以创建与这些复杂体系结构类似的深层内核机器，例如卷积深层内核机器。这样做将为神经网络如何学习提供更好的理论理解，这对可解释的人工智能非常重要，以及执行有监督的学习任务的新的实用算法。参考文献：杨，A，罗伯恩斯，M.，米尔索姆，E.斯库茨，N，和艾奇森，L(2021年)。深度神经网络中的表示学习理论给出了核方法的深度推广。Https://www.ukri.org/what-we-offer/browse-our-areas-of-investment-and-support/artificial-intelligence-technologies/

Deep neural networks are one of the most powerful modern machine learning methods, and can be applied to almost any domain where machine learning is useful, from understanding healthcare data to autonomous and adaptive robotic systems, to smart supply chains, video game design and content creation. The success of modern deep neural networks (DNNs) is based on their ability to use depth to transform the input into high-level representations that are good for solving difficult tasks. This is particularly evident in settings such as transfer learning, where a model is trained on a large scale dataset such as ImageNet, then the top-layer representation is used to solve a related task. However, the manner in which deep networks learn representations across layers is still poorly understood. In particular, perhaps the key theoretical approach for understanding DNN representations is to take an infinite-width limit. These infinite width limits highlight that representations can be formally described using kernels, which are matrices describing the network's judgement about the similarity of all pairs of input examples. However, the issue with infinite-width limits is that they eliminate representation learning; instead kernels become determinstic functions of the inputs, and are not learned based on the targets. Yang et al. (2021) recently introduced a new type of infinite-width limit that also retains representation learning, called the Deep Kernel Machine. It is the first entirely kernel-based deep learning method that gives comparable flexibility to DNNs. Kernel-based methods operate based on the similarity of different data points, which is fundamentally different from the feature-based regime utilised by neural networks, where learning is directly based on the value of data points. This new development shows promise, but the original paper only considers an analogue to simple fully-connected neural networks. Many of the interesting application domains utilise more complicated neural network architectures, such as convolutional neural networks for image tasks, or transformer networks and recurrent networks for sequential tasks like natural language processing. The goal of this project is to develop new extensions to the deep kernel machine literature (and perhaps more generally the deep kernel methods literature) to create deep kernel machine analogues to these complicated architectures, e.g. a convolutional deep kernel machine. Doing so will provide both a better theoretical understanding of how neural networks learn, which is very important for explainable AI, as well as new practical algorithms for performing supervised learning tasks. References: Yang, A., Robeyns, M., Milsom, E., Schoots, N., & Aitchison, L.. (2021). A theory of representation learning in deep neural networks gives a deep generalisation of kernel methods. https://www.ukri.org/what-we-offer/browse-our-areas-of-investment-and-support/artificial-intelligence-technologies/