Disentangling generalization in deep neural networks

尽管深度人工神经网络（ann）在包括视觉和语言建模在内的各种任务上具有出色的经验表现，但允许过度参数化ann记忆训练数据和推广到未见数据的基本原理尚未得到很好的理解。无限宽人工神经网络是通用函数逼近器的标准结果使得许多泛化能力的标准度量在人工神经网络的情况下没有信息。由于无法度量泛化性能（除了在某些数据子集上），或者无法保证训练数据包含一种现象的所有可能示例，使得在安全关键领域（如金融和医疗保健）增加人工神经网络的部署具有风险。该项目将尝试通过分解训练数据、模型架构和训练算法的影响来分析人工神经网络的泛化性能。该项目还将尝试开发比现有措施提供更多信息的概括绩效新措施。该项目将由Varun Kanade教授指导。机器学习研究人员通常会引用模型的“归纳偏差”来解释为什么训练后的模型表达了一个泛化良好的函数，而不是另一个适合训练数据但不泛化的函数。然而，虽然一些归纳偏差很容易定义和理解，比如卷积神经网络中的平移不变性，但其他的则更困难，比如简单前馈架构的明显的简单性偏差。以前的研究倾向于只关注模型的归纳偏差的一个方面，比如架构，而保持其他因素不变，这使得很难确定每个因素对泛化的相对重要性。该项目将涉及对几种不同类型数据的许多架构和优化方法的数学分析，以确定这些因素对训练模型的泛化性能的相对贡献。这将导致理论上驱动的泛化边界的发展，它可以为部署的模型何时按预期执行以及何时可能失败提供指导和保证。该项目的成功完成可能会对提高对人工神经网络的信任产生重大影响，并确保它们的安全部署。该项目属于EPSRC人工智能技术和理论计算机科学研究领域。

Despite the excellent empirical performance of deep artificial neural networks (ANNs) on a wide variety of tasks, including in vision and language modelling, the underlying principles that allow over-parameterised ANNs to both memorise training data and generalise to unseen data are not well understood. A standard result that infinite-width ANNs are universal function approximators renders many standard measures of generalisation ability uninformative in the case of ANNs. The inability to measure generalization performance (except on some subset of the data), or guarantee that the training data contains all possible examples of a phenomenon, makes the increased deployment of ANNs in safety-critical areas, such as finance and healthcare, risky. This project will attempt to analyse the generalisation performance of ANNs by disentangling the effects of the training data, the model architecture, and the training algorithm. The project will also attempt to develop new measures of generalisation performance that are more informative than existing measures. The project will be supervised by Prof. Varun Kanade.Machine learning researchers will often refer to the "inductive bias" of a model to explain why the trained model expresses a function that generalises well, rather than another function that fits the training data but does not generalise. However, while some inductive biases are easy to define and understand, such as translation invariance in convolutional neural networks, others are more difficult, such as the apparent simplicity bias of simple feedforward architectures. Previous studies have tended to focus on only one aspect of the inductive bias of a model, such as the architecture, while holding other factors constant, making it hard to determine the relative importance of each factor on generalisation. This project will involve mathematical analysis of a number of architectures and optimisation methods, on several different types of data, to determine the relative contribution of each of these factors to the generalisation performance of the trained model. This will lead to the development of theoretically motivated generalisation bounds that can provide guidance and guarantees on when deployed models will perform as expected, and when they may fail. The successful completion of this project could have a significant impact on increasing trust in ANNs, and ensuring they are deployed safely.This project falls within the EPSRC artificial intelligence technologies and theoretical computer science research areas.