Disentangling generalization in deep neural networks

尽管深度人工神经网络(ANN)在包括视觉和语言建模在内的各种任务中具有出色的经验表现，但允许过度参数化的ANN既能记忆训练数据又能概括到看不见的数据的基本原理还没有得到很好的理解。无限宽的人工神经网络是通用函数逼近器的标准结果使得许多标准的泛化能力测量在神经网络的情况下不能提供信息。由于无法测量泛化性能(数据的某些子集除外)，也无法保证训练数据包含某一现象的所有可能示例，这使得在金融和医疗保健等安全关键领域增加神经网络的部署具有风险。这个项目将试图通过分离训练数据、模型结构和训练算法的影响来分析神经网络的泛化性能。该项目还将尝试开发比现有衡量标准更具信息量的新的推广绩效衡量标准。该项目将由Varun Kanade.教授监督。机器学习研究人员经常会提到模型的“归纳偏差”，以解释为什么训练的模型表达了一个泛化良好的函数，而不是另一个符合训练数据但不泛化的函数。然而，尽管一些归纳偏差很容易定义和理解，例如卷积神经网络中的平移不变性，但其他一些比较困难，例如简单前馈结构的明显简单性偏差。以前的研究倾向于只关注模型归纳偏差的一个方面，如体系结构，而保持其他因素不变，因此很难确定每个因素对泛化的相对重要性。这个项目将涉及对几种不同类型的数据的一些架构和优化方法的数学分析，以确定这些因素中的每一个对训练模型的泛化性能的相对贡献。这将导致理论上有动机的泛化界限的发展，这些界限可以为部署的模型何时按预期执行以及何时可能失败提供指导和保证。这个项目的成功完成可能会对增加人们对人工神经网络的信任产生重大影响，并确保它们的安全部署。该项目属于EPSRC人工智能技术和理论计算机科学研究领域。

Despite the excellent empirical performance of deep artificial neural networks (ANNs) on a wide variety of tasks, including in vision and language modelling, the underlying principles that allow over-parameterised ANNs to both memorise training data and generalise to unseen data are not well understood. A standard result that infinite-width ANNs are universal function approximators renders many standard measures of generalisation ability uninformative in the case of ANNs. The inability to measure generalization performance (except on some subset of the data), or guarantee that the training data contains all possible examples of a phenomenon, makes the increased deployment of ANNs in safety-critical areas, such as finance and healthcare, risky. This project will attempt to analyse the generalisation performance of ANNs by disentangling the effects of the training data, the model architecture, and the training algorithm. The project will also attempt to develop new measures of generalisation performance that are more informative than existing measures. The project will be supervised by Prof. Varun Kanade.Machine learning researchers will often refer to the "inductive bias" of a model to explain why the trained model expresses a function that generalises well, rather than another function that fits the training data but does not generalise. However, while some inductive biases are easy to define and understand, such as translation invariance in convolutional neural networks, others are more difficult, such as the apparent simplicity bias of simple feedforward architectures. Previous studies have tended to focus on only one aspect of the inductive bias of a model, such as the architecture, while holding other factors constant, making it hard to determine the relative importance of each factor on generalisation. This project will involve mathematical analysis of a number of architectures and optimisation methods, on several different types of data, to determine the relative contribution of each of these factors to the generalisation performance of the trained model. This will lead to the development of theoretically motivated generalisation bounds that can provide guidance and guarantees on when deployed models will perform as expected, and when they may fail. The successful completion of this project could have a significant impact on increasing trust in ANNs, and ensuring they are deployed safely.This project falls within the EPSRC artificial intelligence technologies and theoretical computer science research areas.