CAREER: Towards a Theory of Deep Learning

职业：走向深度学习理论

• 批准号: 2144994
• 负责人: Jason Lee
• 金额: $ 58.3万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-06-01 至 2027-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2144994&HistoricalAwards=false
• 关键词: CAREER; Towards; Theory; Deep; Learning

Over the past decade, deep learning has evolved from conquering research benchmarks to systems that interact with humans on a daily basis, including in machine translation, healthcare, speech recognition, semi-autonomous vehicles, and automation. However, a large gap exists between its empirical successes and a theoretical understanding of why / when it works. This project aims to close this gap through foundational understanding of deep learning and designing algorithms to improve reliability and data efficiency. More broadly, the societal impact of this project include i) theoretical understanding and design of algorithms relevant to machine learning, ii) education plans that develop a new seminar series and workshops for secondary school teachers, and iii) improving disability accommodation in academia.This project is divided into three different thrusts. The first thrust is to understand the algorithmic regularization effect of algorithms and architectures. Using these insights, the team will design better loss functions and architectures to improve accuracy. The second thrust is to theoretically compare the accuracy of networks trained with stochastic gradient descent against their architecture-induced kernel methods. This comparison may theoretically demonstrate that neural networks can do feature learning, which explains the empirical success of deep learning, and that kernel methods cannot. Finally, the project will study representation learning, and theoretically analyze how deep networks can transfer their representations between different domains. Such a transfer will allow a reduction of the labeled data requirements for deep learning, potentially allowing its application to data-starved domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在过去的十年中，深度学习已经从征服研究基准发展成为每天与人类交互的系统，包括机器翻译、医疗保健、语音识别、半自动驾驶汽车和自动化。然而，其实证成功与对其为何/何时有效的理论理解之间存在很大差距。该项目旨在通过对深度学习的基础理解和设计算法来缩小这一差距，以提高可靠性和数据效率。更广泛地说，该项目的社会影响包括 i) 与机器学习相关的算法的理论理解和设计，ii) 为中学教师开发新的研讨会系列和讲习班的教育计划，以及 iii) 改善学术界的残疾人住宿。该项目分为三个不同的主旨。第一个重点是了解算法和架构的算法正则化效果。利用这些见解，团队将设计更好的损失函数和架构以提高准确性。第二个重点是从理论上比较使用随机梯度下降训练的网络与其架构诱导的内核方法的准确性。这种比较可以从理论上证明神经网络可以进行特征学习，这解释了深度学习在经验上的成功，而核方法则不能。最后，该项目将研究表示学习，并从理论上分析深度网络如何在不同领域之间转移其表示。这种转移将减少深度学习的标记数据要求，从而有可能将其应用于数据匮乏的领域。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability

• DOI: 10.48550/arxiv.2209.15594
• 发表时间: 2022-09
• 期刊: ArXiv
• 作者: Alexandru Damian;Eshaan Nichani;Jason D. Lee