Generalisation and expressiveness for over-parameterised neural networks

过度参数化神经网络的泛化和表达能力

• 批准号: 2278529
• 金额: --
• 依托单位: University of Oxford
• 依托单位国家: 英国
• 项目类别: Studentship
• 财政年份: 2019
• 资助国家: 英国
• 起止时间: 2019 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=studentship-2278529
• 关键词: Generalisation; expressiveness; over; parameterised; neural

More than half a century has passed since the mathematical analysis of learning processes truly began, that is when F. Rosenblatt proposed the first model of learning machine, the Perceptron. Since then, huge progresses have been achieved in learning theory, both from theoretical and practical perspectives.However the development of deep neural networks, able to perform better than any theoretical expectation, has underlined the lack of a complete mathematical theory, capable of giving an enlightening insight on the mechanisms underlying the learning process.The basic idea behind learning theory is to find a way to quantify the generalisation capacity of an algorithm, meaning the ability to extrapolate, from a finite and relatively small training dataset, some unknown rule allowing to coherently classify data which are not part of the original training set.A learning algorithm is usually evaluated via the introduction of a loss function, measuring the discrepancy between the correct answer and the answer given by the learning machine. The expectation of this function on the whole set of existing data is called the risk functional. However, in practice there's no way to evaluate it directly, and one has to deal with its approximation evaluated on the training set only, the empirical risk.An efficient learning theory should be able to give tight probability bounds on the difference (generalisation error) between the empirical risk and the risk functional.The Vapnik-Chervonenkis theory, which has been the main direction of theorists' work for about three decades (from the 60's to the 90's), cannot explain the high generalisation capacity of modern deep neural networks, where the machine is able to infer many more parameters than the available training data.Among the theory developed in the last 20 years, of relevant interest is the PAC-Bayes approach, PAC standing for Probably Almost Correct. The main idea in PAC-Bayes is to consider a probability distribution (prior) on the hypothesis space (meaning the space encoding all the possible "classification rules" that the algorithm might choose) which is independent of the training set data. Another distribution (posterior) has to be chosen afterwards and, thanks to a well known result by McAllester, it is then possible to quantify the distance between empirical risk and risk functional in terms of the KL divergence between prior and posterior.In a recent work, Dziugaite and Roy have shown that by optimising a PAC-Bayes bound it is possible to compute nonvacuous numerical bounds for generalisation error.The main goal of this doctoral project is to try to use the PAC-Bayes framework in order to get new theoretical insights allowing for a better understanding of the underlying learning mechanisms in deep neural networks.

自从学习过程的数学分析真正开始以来，已经过去了半个多世纪，也就是F。Rosenblatt提出了学习机的第一个模型，感知器。从那时起，学习理论在理论和实践方面都取得了巨大的进步。然而，深度神经网络的发展，能够比任何理论预期都表现得更好，强调了缺乏完整的数学理论，能够对学习过程的机制提供启发性的见解。学习理论背后的基本思想是找到一种方法来量化概括算法的能力，意味着从有限且相对较小的训练数据集外推某些未知规则的能力，这些规则允许对不属于原始训练集的数据进行连贯分类。通常通过引入损失函数来评估学习算法，测量正确答案与学习机给出的答案之间的差异。该函数对整个现有数据集的期望称为风险泛函。然而，在实践中，没有办法直接评估它，人们必须处理它的近似评估的训练集，经验风险。一个有效的学习理论应该能够给出严格的概率界的差异（泛化误差）之间的经验风险和风险功能。Vapnik-Chervonenkis理论，这是近三十年来理论家们工作的主要方向（从60年代到90年代），无法解释现代深度神经网络的高泛化能力，其中机器能够推断出比可用的训练数据更多的参数。在过去20年发展的理论中，相关的兴趣是PAC-Bayes方法，PAC代表可能几乎正确。PAC-Bayes的主要思想是考虑假设空间（意味着编码算法可能选择的所有可能的“分类规则”的空间）上的概率分布（先验），其独立于训练集数据。另一分布（后验）必须在事后选择，并且由于McAllester的众所周知的结果，然后可以根据先验和后验之间的KL分歧来量化经验风险和风险泛函之间的距离。Dziugaite和Roy已经证明，通过优化PAC-贝叶斯界限是可能的计算非空的数值界限的泛化错误。这个博士项目的主要目标是尝试使用PAC-贝叶斯框架，以获得新的理论见解，从而更好地理解深度神经网络中的底层学习机制。