Collaboration on the Theoretical Foundations of Deep Learning

深度学习的成功已经对工业、商业、科学和社会产生了重大影响。但这项技术的许多方面与经典方法论非常不同，人们对此知之甚少。获得理论上的理解将是克服其缺点的关键。关于深度学习理论基础的合作旨在应对这些挑战：了解支撑深度学习实际成功的数学机制，利用这种理解来阐明当前方法的局限性并将其扩展到目前适用的领域之外，以及启动对出现的一系列数学问题的研究。该团队计划了一系列促进合作的机制，包括电话会议和面对面的研究会议，集中组织的博士后计划，以及博士后和研究生在机构之间访问的计划。合作的研究成果具有直接影响深度学习的许多应用领域的强大潜力。该项目还将通过其教育、人力资源开发和扩大参与计划产生广泛影响，特别是通过使用强调强大导师、灵活性和广泛合作机会的方法培训不同的研究生和博士后；通过一年一度的暑期学校向不同的研究生、博士后和初级教师提供深度学习理论基础的课程；以及通过目标是更广泛地参与合作的研究研讨会和暑期学校。该合作的研究议程建立在以下假设的基础上：过度参数化允许有效的优化；隐式正则化的内插允许泛化；深度通过组合性赋予代表性丰富性。该团队的目标是将这些假设作为一般的数学现象来阐述和严格研究，目的是理解深度学习，扩大其适用性，并开发新的方法。除了能够开发基于原则性设计技术的改进的深度学习方法外，理解深度学习成功的数学机制还将对统计学和数学产生影响，包括对经典统计方法的新观点，如复制核Hilbert空间和决策森林，以及非线性矩阵理论和理解随机景观的新研究方向。此外，合作组织的研究研讨会将向公众开放，并将服务于更广泛的研究社区，以应对这些关键挑战。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The success of deep learning has had a major impact across industry, commerce, science and society. But there are many aspects of this technology that are very different from classical methodology and that are poorly understood. Gaining a theoretical understanding will be crucial for overcoming its drawbacks. The Collaboration on the Theoretical Foundations of Deep Learning aims to address these challenges: understanding the mathematical mechanisms that underpin the practical success of deep learning, using this understanding to elucidate the limitations of current methods and extending them beyond the domains where they are currently applicable, and initiating the study of the array of mathematical problems that emerge. The team has planned a range of mechanisms to facilitate collaboration, including teleconference and in-person research meetings, a centrally organized postdoc program, and a program for visits between institutions by postdocs and graduate students. Research outcomes from the collaboration have strong potential to directly impact the many application domains for deep learning. The project will also have broad impacts through its education, human resource development and broadening participation programs, in particular through training a diverse cohort of graduate students and postdocs using an approach that emphasizes strong mentorship, flexibility, and breadth of collaboration opportunities; through an annual summer school that will deliver curriculum in the theoretical foundations of deep learning to a diverse group of graduate students, postdocs, and junior faculty; and through targeting broader participation in the collaboration’s research workshops and summer schools. The collaboration’s research agenda is built on the following hypotheses: that overparametrization allows efficient optimization; that interpolation with implicit regularization enables generalization; and that depth confers representational richness through compositionality. The team aims to formulate and rigorously study these hypotheses as general mathematical phenomena, with the objective of understanding deep learning, extending its applicability, and developing new methods. Beyond enabling the development of improved deep learning methods based on principled design techniques, understanding the mathematical mechanisms that underlie the success of deep learning will also have repercussions on statistics and mathematics, including a new point of view of classical statistical methods, such as reproducing kernel Hilbert spaces and decision forests, and new research directions in nonlinear matrix theory and in understanding random landscapes. In addition, the research workshops that the collaboration will organize will be open to the public and will serve the broader research community in addressing these key challenges.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.