Generalisation and expressiveness for over-parameterised neural networks

过度参数化神经网络的泛化和表达能力

基本信息

  • 批准号:
    2278529
  • 负责人:
  • 金额:
    --
  • 依托单位:
  • 依托单位国家:
    英国
  • 项目类别:
    Studentship
  • 财政年份:
    2019
  • 资助国家:
    英国
  • 起止时间:
    2019 至 无数据
  • 项目状态:
    已结题

项目摘要

More than half a century has passed since the mathematical analysis of learning processes truly began, that is when F. Rosenblatt proposed the first model of learning machine, the Perceptron. Since then, huge progresses have been achieved in learning theory, both from theoretical and practical perspectives.However the development of deep neural networks, able to perform better than any theoretical expectation, has underlined the lack of a complete mathematical theory, capable of giving an enlightening insight on the mechanisms underlying the learning process.The basic idea behind learning theory is to find a way to quantify the generalisation capacity of an algorithm, meaning the ability to extrapolate, from a finite and relatively small training dataset, some unknown rule allowing to coherently classify data which are not part of the original training set.A learning algorithm is usually evaluated via the introduction of a loss function, measuring the discrepancy between the correct answer and the answer given by the learning machine. The expectation of this function on the whole set of existing data is called the risk functional. However, in practice there's no way to evaluate it directly, and one has to deal with its approximation evaluated on the training set only, the empirical risk.An efficient learning theory should be able to give tight probability bounds on the difference (generalisation error) between the empirical risk and the risk functional.The Vapnik-Chervonenkis theory, which has been the main direction of theorists' work for about three decades (from the 60's to the 90's), cannot explain the high generalisation capacity of modern deep neural networks, where the machine is able to infer many more parameters than the available training data.Among the theory developed in the last 20 years, of relevant interest is the PAC-Bayes approach, PAC standing for Probably Almost Correct. The main idea in PAC-Bayes is to consider a probability distribution (prior) on the hypothesis space (meaning the space encoding all the possible "classification rules" that the algorithm might choose) which is independent of the training set data. Another distribution (posterior) has to be chosen afterwards and, thanks to a well known result by McAllester, it is then possible to quantify the distance between empirical risk and risk functional in terms of the KL divergence between prior and posterior.In a recent work, Dziugaite and Roy have shown that by optimising a PAC-Bayes bound it is possible to compute nonvacuous numerical bounds for generalisation error.The main goal of this doctoral project is to try to use the PAC-Bayes framework in order to get new theoretical insights allowing for a better understanding of the underlying learning mechanisms in deep neural networks.
自从学习过程的数学分析真正开始以来,已经过去了半个多世纪,也就是F。Rosenblatt提出了学习机的第一个模型,感知器。从那时起,学习理论在理论和实践方面都取得了巨大的进步。然而,深度神经网络的发展,能够比任何理论预期都表现得更好,强调了缺乏完整的数学理论,能够对学习过程的机制提供启发性的见解。学习理论背后的基本思想是找到一种方法来量化概括算法的能力,意味着从有限且相对较小的训练数据集外推某些未知规则的能力,这些规则允许对不属于原始训练集的数据进行连贯分类。通常通过引入损失函数来评估学习算法,测量正确答案与学习机给出的答案之间的差异。该函数对整个现有数据集的期望称为风险泛函。然而,在实践中,没有办法直接评估它,人们必须处理它的近似评估的训练集,经验风险。一个有效的学习理论应该能够给出严格的概率界的差异(泛化误差)之间的经验风险和风险功能。Vapnik-Chervonenkis理论,这是近三十年来理论家们工作的主要方向(从60年代到90年代),无法解释现代深度神经网络的高泛化能力,其中机器能够推断出比可用的训练数据更多的参数。在过去20年发展的理论中,相关的兴趣是PAC-Bayes方法,PAC代表可能几乎正确。PAC-Bayes的主要思想是考虑假设空间(意味着编码算法可能选择的所有可能的“分类规则”的空间)上的概率分布(先验),其独立于训练集数据。另一分布(后验)必须在事后选择,并且由于McAllester的众所周知的结果,然后可以根据先验和后验之间的KL分歧来量化经验风险和风险泛函之间的距离。Dziugaite和Roy已经证明,通过优化PAC-贝叶斯界限是可能的计算非空的数值界限的泛化错误。这个博士项目的主要目标是尝试使用PAC-贝叶斯框架,以获得新的理论见解,从而更好地理解深度神经网络中的底层学习机制。

项目成果

期刊论文数量(0)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)

数据更新时间:{{ journalArticles.updateTime }}

{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

数据更新时间:{{ journalArticles.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ monograph.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ sciAawards.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ conferencePapers.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ patent.updateTime }}

其他文献

吉治仁志 他: "トランスジェニックマウスによるTIMP-1の線維化促進機序"最新医学. 55. 1781-1787 (2000)
Hitoshi Yoshiji 等:“转基因小鼠中 TIMP-1 的促纤维化机制”现代医学 55. 1781-1787 (2000)。
  • DOI:
  • 发表时间:
  • 期刊:
  • 影响因子:
    0
  • 作者:
  • 通讯作者:
LiDAR Implementations for Autonomous Vehicle Applications
  • DOI:
  • 发表时间:
    2021
  • 期刊:
  • 影响因子:
    0
  • 作者:
  • 通讯作者:
生命分子工学・海洋生命工学研究室
生物分子工程/海洋生物技术实验室
  • DOI:
  • 发表时间:
  • 期刊:
  • 影响因子:
    0
  • 作者:
  • 通讯作者:
吉治仁志 他: "イラスト医学&サイエンスシリーズ血管の分子医学"羊土社(渋谷正史編). 125 (2000)
Hitoshi Yoshiji 等人:“血管医学与科学系列分子医学图解”Yodosha(涉谷正志编辑)125(2000)。
  • DOI:
  • 发表时间:
  • 期刊:
  • 影响因子:
    0
  • 作者:
  • 通讯作者:
Effect of manidipine hydrochloride,a calcium antagonist,on isoproterenol-induced left ventricular hypertrophy: "Yoshiyama,M.,Takeuchi,K.,Kim,S.,Hanatani,A.,Omura,T.,Toda,I.,Akioka,K.,Teragaki,M.,Iwao,H.and Yoshikawa,J." Jpn Circ J. 62(1). 47-52 (1998)
钙拮抗剂盐酸马尼地平对异丙肾上腺素引起的左心室肥厚的影响:“Yoshiyama,M.,Takeuchi,K.,Kim,S.,Hanatani,A.,Omura,T.,Toda,I.,Akioka,
  • DOI:
  • 发表时间:
  • 期刊:
  • 影响因子:
    0
  • 作者:
  • 通讯作者:

的其他文献

{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

{{ truncateString('', 18)}}的其他基金

An implantable biosensor microsystem for real-time measurement of circulating biomarkers
用于实时测量循环生物标志物的植入式生物传感器微系统
  • 批准号:
    2901954
  • 财政年份:
    2028
  • 资助金额:
    --
  • 项目类别:
    Studentship
Exploiting the polysaccharide breakdown capacity of the human gut microbiome to develop environmentally sustainable dishwashing solutions
利用人类肠道微生物群的多糖分解能力来开发环境可持续的洗碗解决方案
  • 批准号:
    2896097
  • 财政年份:
    2027
  • 资助金额:
    --
  • 项目类别:
    Studentship
A Robot that Swims Through Granular Materials
可以在颗粒材料中游动的机器人
  • 批准号:
    2780268
  • 财政年份:
    2027
  • 资助金额:
    --
  • 项目类别:
    Studentship
Likelihood and impact of severe space weather events on the resilience of nuclear power and safeguards monitoring.
严重空间天气事件对核电和保障监督的恢复力的可能性和影响。
  • 批准号:
    2908918
  • 财政年份:
    2027
  • 资助金额:
    --
  • 项目类别:
    Studentship
Proton, alpha and gamma irradiation assisted stress corrosion cracking: understanding the fuel-stainless steel interface
质子、α 和 γ 辐照辅助应力腐蚀开裂:了解燃料-不锈钢界面
  • 批准号:
    2908693
  • 财政年份:
    2027
  • 资助金额:
    --
  • 项目类别:
    Studentship
Field Assisted Sintering of Nuclear Fuel Simulants
核燃料模拟物的现场辅助烧结
  • 批准号:
    2908917
  • 财政年份:
    2027
  • 资助金额:
    --
  • 项目类别:
    Studentship
Assessment of new fatigue capable titanium alloys for aerospace applications
评估用于航空航天应用的新型抗疲劳钛合金
  • 批准号:
    2879438
  • 财政年份:
    2027
  • 资助金额:
    --
  • 项目类别:
    Studentship
CDT year 1 so TBC in Oct 2024
CDT 第 1 年,预计 2024 年 10 月
  • 批准号:
    2879865
  • 财政年份:
    2027
  • 资助金额:
    --
  • 项目类别:
    Studentship
Developing a 3D printed skin model using a Dextran - Collagen hydrogel to analyse the cellular and epigenetic effects of interleukin-17 inhibitors in
使用右旋糖酐-胶原蛋白水凝胶开发 3D 打印皮肤模型,以分析白细胞介素 17 抑制剂的细胞和表观遗传效应
  • 批准号:
    2890513
  • 财政年份:
    2027
  • 资助金额:
    --
  • 项目类别:
    Studentship
Understanding the interplay between the gut microbiome, behavior and urbanisation in wild birds
了解野生鸟类肠道微生物组、行为和城市化之间的相互作用
  • 批准号:
    2876993
  • 财政年份:
    2027
  • 资助金额:
    --
  • 项目类别:
    Studentship

相似海外基金

III:Small: Expressiveness of Genome Graphs: Construction, Comparison, and Heterogeneity
III:小:基因组图的表现力:构建、比较和异质性
  • 批准号:
    2232121
  • 财政年份:
    2023
  • 资助金额:
    --
  • 项目类别:
    Standard Grant
Comparing the expressiveness of transducers and simply-typed linear lambda-calculi
比较换能器和简单类型线性 lambda 演算的表达能力
  • 批准号:
    2865040
  • 财政年份:
    2023
  • 资助金额:
    --
  • 项目类别:
    Studentship
Can AI Rakugoka entertain people? -Improved expressiveness of rakugo speech synthesis and automatic generation of storytelling
AI落语可以娱乐人们吗?
  • 批准号:
    21K19808
  • 财政年份:
    2021
  • 资助金额:
    --
  • 项目类别:
    Grant-in-Aid for Challenging Research (Exploratory)
Curriculum development based on the cooperation between kindergarten,elementary school,and junior highschool; and international comparison for the development of logical thinking and expressiveness
幼儿园、小学、初中合作的课程开发;
  • 批准号:
    20K02775
  • 财政年份:
    2020
  • 资助金额:
    --
  • 项目类别:
    Grant-in-Aid for Scientific Research (C)
Efforts to Improve Competence and Emotional Expressiveness in Expression Activities of Preschool Children through Singing-Instruction
通过歌唱教学提高学龄前儿童表达能力和情感表达能力
  • 批准号:
    19K02665
  • 财政年份:
    2019
  • 资助金额:
    --
  • 项目类别:
    Grant-in-Aid for Scientific Research (C)
Linguistic features of emotional expressions in English and teaching methods for improving English emotional expressiveness
英语情感表达的语言特点及提高英语情感表达的教学方法
  • 批准号:
    16K02932
  • 财政年份:
    2016
  • 资助金额:
    --
  • 项目类别:
    Grant-in-Aid for Scientific Research (C)
Dynamic Expressiveness of Logics
逻辑的动态表达
  • 批准号:
    228818952
  • 财政年份:
    2013
  • 资助金额:
    --
  • 项目类别:
    Research Grants
RI: Small: Expressiveness and Automated Bundling in Mechanism Design: Principles and Computational Methodologies
RI:小:机制设计中的表现力和自动捆绑:原理和计算方法
  • 批准号:
    1320620
  • 财政年份:
    2013
  • 资助金额:
    --
  • 项目类别:
    Standard Grant
SHF: Small: Collaborative Research: Balancing Expressiveness and Modular Reasoning for Aspect-Oriented Programming
SHF:小型:协作研究:平衡面向方面编程的表达性和模块化推理
  • 批准号:
    1017262
  • 财政年份:
    2010
  • 资助金额:
    --
  • 项目类别:
    Continuing Grant
SHF: Small: Collaborative Research: Balancing Expressiveness and Modular Reasoning for Aspect-oriented Programming
SHF:小型:协作研究:平衡面向方面编程的表达性和模块化推理
  • 批准号:
    1017334
  • 财政年份:
    2010
  • 资助金额:
    --
  • 项目类别:
    Continuing Grant
{{ showInfoDetail.title }}

作者:{{ showInfoDetail.author }}

知道了