Collaboration on the Theoretical Foundations of Deep Learning

深度学习的成功对工业、商业、科学和社会都产生了重大影响。但是这项技术的许多方面与传统的方法有很大的不同，人们对它们的理解也很差。从理论上理解它对于克服它的缺点是至关重要的。深度学习理论基础合作旨在解决这些挑战：理解支撑深度学习实践成功的数学机制，利用这种理解来阐明当前方法的局限性，并将其扩展到当前适用的领域之外，并启动对出现的一系列数学问题的研究。该团队计划了一系列促进合作的机制，包括电话会议和面对面的研究会议，一个集中组织的博士后计划，以及博士后和研究生在机构之间访问的计划。合作的研究成果有很大的潜力直接影响深度学习的许多应用领域。该项目还将通过其教育、人力资源开发和扩大参与计划产生广泛影响，特别是通过使用强调强大的指导、灵活性和广泛合作机会的方法培训多样化的研究生和博士后；通过一年一度的暑期学校，为研究生、博士后和初级教师提供深度学习理论基础课程；通过扩大合作研究研讨会和暑期学校的参与范围。合作的研究议程是建立在以下假设：过度参数化允许有效的优化；隐式正则化的插值实现了泛化；这种深度通过构图赋予了表现的丰富性。该团队的目标是将这些假设作为一般的数学现象来制定和严格研究，目的是理解深度学习，扩展其适用性，并开发新的方法。除了能够基于原则设计技术开发改进的深度学习方法之外，理解深度学习成功背后的数学机制也将对统计学和数学产生影响，包括对经典统计方法的新观点，例如再现核希尔伯特空间和决策森林，以及非线性矩阵理论和理解随机景观的新研究方向。此外，合作将组织的研究研讨会将向公众开放，并将在解决这些关键挑战方面为更广泛的研究界服务。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The success of deep learning has had a major impact across industry, commerce, science and society. But there are many aspects of this technology that are very different from classical methodology and that are poorly understood. Gaining a theoretical understanding will be crucial for overcoming its drawbacks. The Collaboration on the Theoretical Foundations of Deep Learning aims to address these challenges: understanding the mathematical mechanisms that underpin the practical success of deep learning, using this understanding to elucidate the limitations of current methods and extending them beyond the domains where they are currently applicable, and initiating the study of the array of mathematical problems that emerge. The team has planned a range of mechanisms to facilitate collaboration, including teleconference and in-person research meetings, a centrally organized postdoc program, and a program for visits between institutions by postdocs and graduate students. Research outcomes from the collaboration have strong potential to directly impact the many application domains for deep learning. The project will also have broad impacts through its education, human resource development and broadening participation programs, in particular through training a diverse cohort of graduate students and postdocs using an approach that emphasizes strong mentorship, flexibility, and breadth of collaboration opportunities; through an annual summer school that will deliver curriculum in the theoretical foundations of deep learning to a diverse group of graduate students, postdocs, and junior faculty; and through targeting broader participation in the collaboration’s research workshops and summer schools. The collaboration’s research agenda is built on the following hypotheses: that overparametrization allows efficient optimization; that interpolation with implicit regularization enables generalization; and that depth confers representational richness through compositionality. The team aims to formulate and rigorously study these hypotheses as general mathematical phenomena, with the objective of understanding deep learning, extending its applicability, and developing new methods. Beyond enabling the development of improved deep learning methods based on principled design techniques, understanding the mathematical mechanisms that underlie the success of deep learning will also have repercussions on statistics and mathematics, including a new point of view of classical statistical methods, such as reproducing kernel Hilbert spaces and decision forests, and new research directions in nonlinear matrix theory and in understanding random landscapes. In addition, the research workshops that the collaboration will organize will be open to the public and will serve the broader research community in addressing these key challenges.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.