Disentangling generalization in deep neural networks

点击翻译按钮获取中文摘要

Despite the excellent empirical performance of deep artificial neural networks (ANNs) on a wide variety of tasks, including in vision and language modelling, the underlying principles that allow over-parameterised ANNs to both memorise training data and generalise to unseen data are not well understood. A standard result that infinite-width ANNs are universal function approximators renders many standard measures of generalisation ability uninformative in the case of ANNs. The inability to measure generalization performance (except on some subset of the data), or guarantee that the training data contains all possible examples of a phenomenon, makes the increased deployment of ANNs in safety-critical areas, such as finance and healthcare, risky. This project will attempt to analyse the generalisation performance of ANNs by disentangling the effects of the training data, the model architecture, and the training algorithm. The project will also attempt to develop new measures of generalisation performance that are more informative than existing measures. The project will be supervised by Prof. Varun Kanade.Machine learning researchers will often refer to the "inductive bias" of a model to explain why the trained model expresses a function that generalises well, rather than another function that fits the training data but does not generalise. However, while some inductive biases are easy to define and understand, such as translation invariance in convolutional neural networks, others are more difficult, such as the apparent simplicity bias of simple feedforward architectures. Previous studies have tended to focus on only one aspect of the inductive bias of a model, such as the architecture, while holding other factors constant, making it hard to determine the relative importance of each factor on generalisation. This project will involve mathematical analysis of a number of architectures and optimisation methods, on several different types of data, to determine the relative contribution of each of these factors to the generalisation performance of the trained model. This will lead to the development of theoretically motivated generalisation bounds that can provide guidance and guarantees on when deployed models will perform as expected, and when they may fail. The successful completion of this project could have a significant impact on increasing trust in ANNs, and ensuring they are deployed safely.This project falls within the EPSRC artificial intelligence technologies and theoretical computer science research areas.