Collaboration on the Theoretical Foundations of Deep Learning

深度学习理论基础的合作

• 批准号: 2031883
• 负责人: Peter Bartlett
• 金额: $ 500万
• 依托单位: University of California-Berkeley
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2031883&HistoricalAwards=false
• 关键词: Collaboration; Theoretical; Foundations; Deep; Learning

The success of deep learning has had a major impact across industry, commerce, science and society. But there are many aspects of this technology that are very different from classical methodology and that are poorly understood. Gaining a theoretical understanding will be crucial for overcoming its drawbacks. The Collaboration on the Theoretical Foundations of Deep Learning aims to address these challenges: understanding the mathematical mechanisms that underpin the practical success of deep learning, using this understanding to elucidate the limitations of current methods and extending them beyond the domains where they are currently applicable, and initiating the study of the array of mathematical problems that emerge. The team has planned a range of mechanisms to facilitate collaboration, including teleconference and in-person research meetings, a centrally organized postdoc program, and a program for visits between institutions by postdocs and graduate students. Research outcomes from the collaboration have strong potential to directly impact the many application domains for deep learning. The project will also have broad impacts through its education, human resource development and broadening participation programs, in particular through training a diverse cohort of graduate students and postdocs using an approach that emphasizes strong mentorship, flexibility, and breadth of collaboration opportunities; through an annual summer school that will deliver curriculum in the theoretical foundations of deep learning to a diverse group of graduate students, postdocs, and junior faculty; and through targeting broader participation in the collaboration’s research workshops and summer schools. The collaboration’s research agenda is built on the following hypotheses: that overparametrization allows efficient optimization; that interpolation with implicit regularization enables generalization; and that depth confers representational richness through compositionality. The team aims to formulate and rigorously study these hypotheses as general mathematical phenomena, with the objective of understanding deep learning, extending its applicability, and developing new methods. Beyond enabling the development of improved deep learning methods based on principled design techniques, understanding the mathematical mechanisms that underlie the success of deep learning will also have repercussions on statistics and mathematics, including a new point of view of classical statistical methods, such as reproducing kernel Hilbert spaces and decision forests, and new research directions in nonlinear matrix theory and in understanding random landscapes. In addition, the research workshops that the collaboration will organize will be open to the public and will serve the broader research community in addressing these key challenges.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习的成功对工业、商业、科学和社会产生了重大影响。但是，这项技术的许多方面与经典方法学有很大的不同，而且人们对它的理解也很有限。获得理论上的理解将是克服其缺点的关键。 深度学习理论基础合作项目旨在解决这些挑战：理解支撑深度学习实际成功的数学机制，利用这种理解来阐明当前方法的局限性，并将其扩展到当前适用的领域之外，并开始研究出现的一系列数学问题。 该团队计划了一系列促进合作的机制，包括电话会议和面对面的研究会议，一个集中组织的博士后计划，以及博士后和研究生机构之间的访问计划。 合作的研究成果有很大的潜力直接影响深度学习的许多应用领域。该项目还将通过其教育，人力资源开发和扩大参与计划产生广泛影响，特别是通过使用强调强有力的指导，灵活性和广泛合作机会的方法培训多样化的研究生和博士后;通过一年一度的暑期学校，将提供深度学习的理论基础课程，以不同的研究生群体，博士后和初级教师;并通过在合作的研究讲习班和暑期学校的目标更广泛的参与。合作的研究议程是建立在以下假设：overparametrization允许有效的优化;插值与隐式正则化使泛化;和深度赋予代表丰富性通过组合。该团队的目标是将这些假设作为一般的数学现象来制定和严格研究，目的是理解深度学习，扩展其适用性并开发新方法。 除了基于原则性设计技术开发改进的深度学习方法外，理解深度学习成功背后的数学机制也将对统计学和数学产生影响，包括经典统计方法的新观点，如再生核希尔伯特空间和决策森林，以及非线性矩阵理论和理解随机景观的新研究方向。 此外，合作组织的研究研讨会将向公众开放，并将服务于更广泛的研究社区，以解决这些关键挑战。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估的支持。

项目成果

Covariance’s Loss is Privacy’s Gain: Computationally Efficient, Private and Accurate Synthetic Data

• DOI: 10.1007/s10208-022-09591-7
• 发表时间: 2021-07
• 期刊: Foundations of Computational Mathematics
• 影响因子: 3
• 作者: M. Boedihardjo;T. Strohmer;R. Vershynin

Partial recovery and weak consistency in the non-uniform hypergraph Stochastic Block Model

非均匀超图随机块模型中的部分恢复和弱一致性

• 发表时间: 2021
• 期刊: ArXivorg
• 作者: Dumitriu, Ioana;Wang, Haixiao;Zhu, Yizhe
• 通讯作者: Zhu, Yizhe

Gradient dynamics of single-neuron autoencoders on orthogonal data

正交数据上单神经元自动编码器的梯度动力学

• 发表时间: 2022
• 期刊: 14th Annual Workshop on Optimization for Machine Learning (NeurIPS 2022 Workshop
• 作者: Ghosh, Nikhil;Frei, Spencer;Ha, Wooseok;Yu, Bin
• 通讯作者: Yu, Bin

Proof of the satisfiability conjecture for large $k$

大 $k$ 的可满足性猜想的证明

• DOI: 10.4007/annals.2022.196.1.1
• 发表时间: 2022
• 期刊: Annals of Mathematics
• 影响因子: 4.9
• 作者: Ding, Jian;Sly, Allan;Sun, Nike
• 通讯作者: Sun, Nike

The interplay between implicit bias and benign overfitting in two-layer linear networks

两层线性网络中隐式偏差和良性过度拟合之间的相互作用

• 发表时间: 2022
• 期刊: Journal of machine learning research
• 影响因子: 6
• 作者: Chatterji, Niladri S.;Long, Philip M.;Bartlett, Peter L.
• 通讯作者: Bartlett, Peter L.