CAREER: Theoretical foundations of neural networks - representation, optimization, and generalization

神经网络构成了机器学习的最新进展和突然普及的支柱。然而，尽管取得了广泛的经验进展，但对他们的行为仍缺乏令人满意的理解。随着神经网络越来越多地进入面向人类的服务(自动驾驶汽车、医疗诊断等)，这种现状，特别是其安全分支变得令人担忧。本项目旨在从理论上理解神经网络的基础，分为三部分：(A)关于哪些现象可以被神经网络简洁地近似的表示问题；(B)如何有效地使神经网络与数据相匹配的优化问题；以及(C)关于为什么神经网络不仅可以拟合它们已经看到的数据，而且还可以拟合它们没有看到的数据的泛化问题。发展这一认识将构成该项目三个更广泛影响的核心：(1)研究部分旨在提高面向用户的神经网络部署的安全性和可靠性；(2)作为教育部分，研究将得到简化，并纳入免费提供的课程笔记；(3)该奖项支持由PI共同创立的两项外展努力：UIUC-ML，一个大学范围内的ML研讨会；以及Midwest ML研讨会，一个中西部ML年度聚会。核心的表示问题是：神经网络表示的特殊之处在哪里？更详细地说，所提出的表示问题首先是表征通过向网络添加单层所获得的能量，其次是表征递归神经网络的表示性质，即随着所消耗的时间序列而演化其状态的神经网络。接下来是优化的话题，其中的关键之谜是神经网络如何设法用简单的迭代下降方案完美地匹配他们的数据，尽管问题显然是非凸性的。这里的计划是建立一个更强大的性质：这些迭代方案设法输出网络，这些网络不仅符合他们的数据，而且按照经典的边际理论，这样做是自信的。最后，提案以概括的主题结束。第一个目标是开发精确的泛化界限，以使它们可以通过有效的正则化方案在算法上强制执行，第二个目标是将这些技术应用于将神经网络适配到概率分布，特别是训练生成性对抗网络的问题。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Neural networks form the backbone of machine learning's recent advances and sudden ubiquity. Despite this extensive empirical progress, however, a satisfactory understanding of their behavior is still missing. As neural networks enter more and more into human-facing services (self-driving cars, medical diagnostics, etc.), this status quo and in particular its safety ramifications becomes worrisome. This project aims for a theoretical understanding of the foundations of neural networks, divided into three pieces: (a) the representation question regarding which phenomena can be succinctly approximated by neural networks; (b) the optimization question of how to efficiently fit neural networks to data; and (c) the generalization question on why neural networks can fit not only the data they have seen but also the data they have not seen. Developing this understanding will form the core of this project's three broader impacts: (1) the research component will aim to improve safety and reliability of user-facing deployments of neural networks; (2) as an educational component, the research will be simplified and incorporated into freely available course notes; (3) the award supports two outreach efforts co-founded by the PI: UIUC-ML, a university-wide ML seminar; and the midwest ML symposium, a yearly midwest ML gathering.In more detail, the technical focus of this project, divided into the three learning theoretic topics above, is as follows. The core representation question is: what makes neural network representation special? In more detail, the proposed representation questions are firstly to characterize the power gained by adding a single layer to a network, and secondly to characterize the representation properties of recurrent neural networks, namely neural networks which evolve their state along with a time series they consume. Next comes the topic of optimization, where the key mystery is how neural networks manage to perfectly fit their data with simple iterative descent schemes, despite the apparent nonconvexity of the problem. The plan here is to establish an even stronger property: these iterative schemes manage to output networks which not only fit their data, but do so confidently, in the classical sense of margin theory. Finally, the proposal closes with the topic of generalization. The first goal is to develop refined generalization bounds to the point that they can be algorithmically enforced via effective regularization schemes, and secondarily to apply these techniques to the fitting of neural networks to probability distributions, specifically the problem of training Generative Adversarial Networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.