CAREER: Towards a Theory of Deep Learning

职业：走向深度学习理论

• 批准号: 2144994
• 负责人: Jason Lee
• 金额: $ 58.3万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-06-01 至 2027-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2144994&HistoricalAwards=false
• 关键词: CAREER; Towards; Theory; Deep; Learning

Over the past decade, deep learning has evolved from conquering research benchmarks to systems that interact with humans on a daily basis, including in machine translation, healthcare, speech recognition, semi-autonomous vehicles, and automation. However, a large gap exists between its empirical successes and a theoretical understanding of why / when it works. This project aims to close this gap through foundational understanding of deep learning and designing algorithms to improve reliability and data efficiency. More broadly, the societal impact of this project include i) theoretical understanding and design of algorithms relevant to machine learning, ii) education plans that develop a new seminar series and workshops for secondary school teachers, and iii) improving disability accommodation in academia.This project is divided into three different thrusts. The first thrust is to understand the algorithmic regularization effect of algorithms and architectures. Using these insights, the team will design better loss functions and architectures to improve accuracy. The second thrust is to theoretically compare the accuracy of networks trained with stochastic gradient descent against their architecture-induced kernel methods. This comparison may theoretically demonstrate that neural networks can do feature learning, which explains the empirical success of deep learning, and that kernel methods cannot. Finally, the project will study representation learning, and theoretically analyze how deep networks can transfer their representations between different domains. Such a transfer will allow a reduction of the labeled data requirements for deep learning, potentially allowing its application to data-starved domains.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在过去的十年中，深度学习已经从征服研究基准发展到每天与人类交互的系统，包括机器翻译，医疗保健，语音识别，半自动驾驶汽车和自动化。然而，它的经验成功与理论上对它为什么/何时起作用的理解之间存在着很大的差距。该项目旨在通过对深度学习的基础理解和设计算法来缩小这一差距，以提高可靠性和数据效率。更广泛地说，该项目的社会影响包括i）机器学习相关算法的理论理解和设计，ii）为中学教师开发新的研讨会系列和讲习班的教育计划，以及iii）改善学术界的残疾人住宿。该项目分为三个不同的重点。第一个目标是理解算法和架构的算法正则化效果。利用这些见解，该团队将设计更好的损失函数和架构，以提高准确性。第二个目标是从理论上比较用随机梯度下降训练的网络与其结构诱导的核方法的准确性。这种比较可以从理论上证明神经网络可以进行特征学习，这解释了深度学习的经验成功，而内核方法不能。最后，该项目将研究表征学习，并从理论上分析深度网络如何在不同域之间传输表征。这样的转移将减少深度学习的标记数据要求，从而可能使其应用于数据匮乏的领域。该奖项反映了NSF的法定使命，并且通过使用基金会的知识价值和更广泛的知识价值进行评估，被认为值得支持影响审查标准。

项目成果

Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability

• DOI: 10.48550/arxiv.2209.15594
• 发表时间: 2022-09
• 期刊: ArXiv
• 作者: Alexandru Damian;Eshaan Nichani;Jason D. Lee