Inductive bias selection in Bayesian models

贝叶斯模型中的归纳偏差选择

• 批准号: 2740634
• 金额: --
• 依托单位: University of Oxford
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2022
• 资助国家: 英国
• 起止时间: 2022 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2740634
• 关键词: Inductive; bias; selection; Bayesian; models

This project falls within the EPSRC Mathematical Sciences research area.Over the past few years, machine learning models, with neural networks at the forefront, have achieved state-of-the-art performance in a variety of tasks. Among the factors that have contributed to such breakthroughs are significant architectural innovations. Depending on the data one wants to model and the task one wants to solve, extremely specialised models have been developed. For instance, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and graph neural networks (GNNs) are especially suited for image, sequential, and graph data, respectively. Compared to more general models, such as multi-layer perceptrons (MLPs), these specialised models are much more effective in modelling the data they are designed for, despite not necessarily possessing a higher capacity. In other words, while all these models might have the ability to fit the training data equally well, they differ enormously in how they generalise to unseen data. Such a difference in generalisation performance is due to the different sets of assumptions about how the data points relate to one another, called inductive biases, implicitly encoded in the models' architecture. For example, the way in which the architecture of CNNs is organised encourages them to be translation invariant: images with the same pattern in different positions will result in the same output.A downside of highly specialised models with powerful inductive biases is that they currently require human supervision and domain knowledge to design. In particular, candidate models are usually evaluated with cross-validation until a satisfactory solution is found. As the space of candidate architectures is usually huge, such a process often becomes extremely expensive and time-consuming. Conversely, a recent research direction, pursued by Prof. Van der Wilk and his group, is to use ideas from Bayesian model selection to design training objectives amenable to gradient-based optimisation to simultaneously learn the model's architecture and its parameters. This approach has the potential to significantly streamline the development of task-specific architectures by automating the model design pipeline.During my PhD, I will further investigate automatic inductive bias selection in Bayesian models. In this context, I will study Gaussian processes and Bayesian neural networks, as they represent flexible models that can be easily made to incorporate a wide array of inductive biases. The project will involve designing model parameterisations and training objectives suitable for this task. At the same time, I will strive to combine automatic inductive bias selection with other desirable features of Bayesian modelling, such as uncertainty quantification. To start, I will explore the inductive biases useful for modelling dynamical systems, with the aim of developing robust and scalable Bayesian models with potential applications ranging from the natural sciences to engineering and finance.

该项目属于 EPSRC 数学科学研究领域。在过去的几年里，以神经网络为前沿的机器学习模型在各种任务中都取得了最先进的性能。促成这些突破的因素包括重大的架构创新。根据想要建模的数据和想要解决的任务，已经开发出了极其专业的模型。例如，卷积神经网络 (CNN)、循环神经网络 (RNN) 和图神经网络 (GNN) 分别特别适合图像、序列和图数据。与更通用的模型（例如多层感知器（MLP））相比，这些专用模型在对其设计的数据进行建模时要有效得多，尽管不一定具有更高的容量。换句话说，虽然所有这些模型都可能具有同样好的拟合训练数据的能力，但它们在泛化到未见过的数据的方式上存在巨大差异。泛化性能的这种差异是由于关于数据点如何相互关联的不同假设造成的，称为归纳偏差，隐式编码在模型的架构中。例如，CNN 架构的组织方式鼓励它们具有平移不变性：不同位置具有相同模式的图像将产生相同的输出。具有强大归纳偏差的高度专业化模型的一个缺点是，它们目前需要人类监督和领域知识来设计。特别是，通常通过交叉验证来评估候选模型，直到找到满意的解决方案。由于候选架构的空间通常很大，因此这样的过程通常变得极其昂贵且耗时。相反，Van der Wilk 教授及其团队最近追求的一个研究方向是利用贝叶斯模型选择的思想来设计适合基于梯度的优化的训练目标，以同时学习模型的架构及其参数。这种方法有可能通过自动化模型设计流程来显着简化特定于任务的架构的开发。在攻读博士学位期间，我将进一步研究贝叶斯模型中的自动归纳偏差选择。在这种情况下，我将研究高斯过程和贝叶斯神经网络，因为它们代表了灵活的模型，可以轻松地纳入各种归纳偏差。该项目将涉及设计适合该任务的模型参数化和训练目标。同时，我将努力将自动归纳偏差选择与贝叶斯建模的其他理想功能（例如不确定性量化）结合起来。首先，我将探索可用于动态系统建模的归纳偏差，目的是开发稳健且可扩展的贝叶斯模型，其潜在应用范围包括自然科学、工程和金融。