CAREER: Theoretical foundations of neural networks - representation, optimization, and generalization

神经网络构成了机器学习最近的进步和突然普及的支柱。然而，尽管取得了广泛的经验进展，对它们的行为仍缺乏令人满意的理解。随着神经网络越来越多地进入面向人类的服务（自动驾驶汽车、医疗诊断等），这种现状，尤其是其安全后果变得令人担忧。该项目旨在从理论上理解神经网络的基础，分为三个部分：(a)表征问题，关于哪些现象可以用神经网络简洁地近似；(b)如何有效地将神经网络与数据拟合的优化问题；(c)泛化问题，即为什么神经网络不仅可以拟合它们看到的数据，也可以拟合它们没有看到的数据。发展这种理解将构成该项目三个更广泛影响的核心：(1)研究部分将旨在提高面向用户的神经网络部署的安全性和可靠性；(2)作为教育组成部分，研究将被简化并纳入免费提供的课程笔记中；(3)该奖项支持PI共同创立的两项推广工作：UIUC-ML，一个大学范围内的ML研讨会；以及中西部ML研讨会，这是一年一度的中西部ML聚会。更详细地说，本项目的技术重点分为上述三个学习理论课题，具体如下：表征的核心问题是：是什么让神经网络表征变得特别？更详细地说，所提出的表征问题首先是表征通过向网络中添加单层所获得的功率，其次是表征递归神经网络的表征特性，即随着它们消耗的时间序列而进化其状态的神经网络。接下来是优化的主题，其中关键的奥秘是神经网络如何设法用简单的迭代下降方案完美地拟合他们的数据，尽管问题的明显非凸性。这里的计划是建立一个更强的性质：这些迭代方案设法输出网络，不仅适合他们的数据，而且自信地做到这一点，在经典意义上的边际理论。最后，该提案以泛化主题结束。第一个目标是开发精细的泛化边界，使其可以通过有效的正则化方案在算法上强制执行，其次是将这些技术应用于神经网络对概率分布的拟合，特别是训练生成对抗网络的问题。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Neural networks form the backbone of machine learning's recent advances and sudden ubiquity. Despite this extensive empirical progress, however, a satisfactory understanding of their behavior is still missing. As neural networks enter more and more into human-facing services (self-driving cars, medical diagnostics, etc.), this status quo and in particular its safety ramifications becomes worrisome. This project aims for a theoretical understanding of the foundations of neural networks, divided into three pieces: (a) the representation question regarding which phenomena can be succinctly approximated by neural networks; (b) the optimization question of how to efficiently fit neural networks to data; and (c) the generalization question on why neural networks can fit not only the data they have seen but also the data they have not seen. Developing this understanding will form the core of this project's three broader impacts: (1) the research component will aim to improve safety and reliability of user-facing deployments of neural networks; (2) as an educational component, the research will be simplified and incorporated into freely available course notes; (3) the award supports two outreach efforts co-founded by the PI: UIUC-ML, a university-wide ML seminar; and the midwest ML symposium, a yearly midwest ML gathering.In more detail, the technical focus of this project, divided into the three learning theoretic topics above, is as follows. The core representation question is: what makes neural network representation special? In more detail, the proposed representation questions are firstly to characterize the power gained by adding a single layer to a network, and secondly to characterize the representation properties of recurrent neural networks, namely neural networks which evolve their state along with a time series they consume. Next comes the topic of optimization, where the key mystery is how neural networks manage to perfectly fit their data with simple iterative descent schemes, despite the apparent nonconvexity of the problem. The plan here is to establish an even stronger property: these iterative schemes manage to output networks which not only fit their data, but do so confidently, in the classical sense of margin theory. Finally, the proposal closes with the topic of generalization. The first goal is to develop refined generalization bounds to the point that they can be algorithmically enforced via effective regularization schemes, and secondarily to apply these techniques to the fitting of neural networks to probability distributions, specifically the problem of training Generative Adversarial Networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.