Inductive bias selection in Bayesian models

该项目属于EPSRC数学科学研究领域。在过去的几年里，以神经网络为前沿的机器学习模型在各种任务中取得了最先进的性能。促成这些突破的因素之一是重大的建筑创新。根据想要建模的数据和想要解决的任务，已经开发出了非常专业的模型。例如，卷积神经网络（cnn）、循环神经网络（rnn）和图神经网络（gnn）分别特别适合于图像、序列和图数据。与更一般的模型（如多层感知器（mlp））相比，这些专门的模型在为其设计的数据建模方面更有效，尽管不一定具有更高的容量。换句话说，虽然所有这些模型可能都有能力很好地拟合训练数据，但它们在如何推广到看不见的数据方面存在巨大差异。这种泛化性能的差异是由于关于数据点如何相互关联的不同假设集，称为归纳偏差，隐式编码在模型的架构中。例如，cnn架构的组织方式鼓励它们具有平移不变性：在不同位置具有相同模式的图像将导致相同的输出。具有强大归纳偏差的高度专业化模型的一个缺点是，它们目前需要人类监督和领域知识来设计。特别是，候选模型通常通过交叉验证进行评估，直到找到令人满意的解决方案。由于候选体系结构的空间通常很大，这样的过程通常变得非常昂贵和耗时。相反，Van der Wilk教授和他的团队最近追求的一个研究方向是使用贝叶斯模型选择的思想来设计适合基于梯度的优化的训练目标，同时学习模型的架构和参数。这种方法有可能通过自动化模型设计管道来显著地简化特定于任务的体系结构的开发。在博士期间，我将进一步研究贝叶斯模型中的自动归纳偏差选择。在这种情况下，我将研究高斯过程和贝叶斯神经网络，因为它们代表了灵活的模型，可以很容易地将各种归纳偏差结合起来。该项目将包括设计适合此任务的模型参数化和训练目标。同时，我将努力将自动归纳偏置选择与贝叶斯建模的其他可取特性（如不确定性量化）结合起来。首先，我将探讨对动态系统建模有用的归纳偏差，目的是开发具有从自然科学到工程和金融等潜在应用的鲁棒和可扩展贝叶斯模型。

This project falls within the EPSRC Mathematical Sciences research area.Over the past few years, machine learning models, with neural networks at the forefront, have achieved state-of-the-art performance in a variety of tasks. Among the factors that have contributed to such breakthroughs are significant architectural innovations. Depending on the data one wants to model and the task one wants to solve, extremely specialised models have been developed. For instance, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and graph neural networks (GNNs) are especially suited for image, sequential, and graph data, respectively. Compared to more general models, such as multi-layer perceptrons (MLPs), these specialised models are much more effective in modelling the data they are designed for, despite not necessarily possessing a higher capacity. In other words, while all these models might have the ability to fit the training data equally well, they differ enormously in how they generalise to unseen data. Such a difference in generalisation performance is due to the different sets of assumptions about how the data points relate to one another, called inductive biases, implicitly encoded in the models' architecture. For example, the way in which the architecture of CNNs is organised encourages them to be translation invariant: images with the same pattern in different positions will result in the same output.A downside of highly specialised models with powerful inductive biases is that they currently require human supervision and domain knowledge to design. In particular, candidate models are usually evaluated with cross-validation until a satisfactory solution is found. As the space of candidate architectures is usually huge, such a process often becomes extremely expensive and time-consuming. Conversely, a recent research direction, pursued by Prof. Van der Wilk and his group, is to use ideas from Bayesian model selection to design training objectives amenable to gradient-based optimisation to simultaneously learn the model's architecture and its parameters. This approach has the potential to significantly streamline the development of task-specific architectures by automating the model design pipeline.During my PhD, I will further investigate automatic inductive bias selection in Bayesian models. In this context, I will study Gaussian processes and Bayesian neural networks, as they represent flexible models that can be easily made to incorporate a wide array of inductive biases. The project will involve designing model parameterisations and training objectives suitable for this task. At the same time, I will strive to combine automatic inductive bias selection with other desirable features of Bayesian modelling, such as uncertainty quantification. To start, I will explore the inductive biases useful for modelling dynamical systems, with the aim of developing robust and scalable Bayesian models with potential applications ranging from the natural sciences to engineering and finance.