Collaborative Research: Transferable, Hierarchical, Expressive, Optimal, Robust, Interpretable Networks

深度学习的最新进展催生了许多颠覆性技术：从自动语音识别系统到自动化超市，再到自动驾驶汽车。然而，深度网络的复杂性和大规模性质使得它们很难分析，因此，它们大多被用作黑匣子，没有对其性能的正式保证。例如，深度网络提供了自我报告的信心得分，但它们经常不准确和未校准，或者在极少数情况下可能会犯下大错误。此外，深层网络的设计仍然是一门艺术，在很大程度上是由数据集的经验表现推动的。随着深度学习系统越来越多地应用于我们的日常生活中，理解他们的预测是否满足某些期望的性质变得至关重要。NSF-Simons在深度学习的数学和科学基础上的研究合作的目标是开发一个数学、统计和计算框架，帮助解释当前网络架构的成功，了解其陷阱，并以保证的信心、健壮性、可解释性、最佳性和可转移性来指导新型架构的设计。该项目将通过创建数据科学基础方面的新本科生和研究生课程，并组织一系列合作研究活动，包括学期研究项目和深度学习基础暑期学校，培养具有数据科学技能的多元化STEM员工队伍，这些技能对美国经济的全球竞争力至关重要。该项目还将通过让本科生参与数据科学基础，对妇女和代表性不足的少数群体产生影响。深入的网络大大改善了模式识别系统的性能。然而，这种成功的数学原因仍然难以捉摸。例如，不清楚为什么深层网络泛化或转移到新的任务，或者为什么简单的优化策略可以达到相关非凸优化问题的局部或全局最小值。此外，没有原则性的方法来设计网络的体系结构以使其满足某些期望的特性，如可表现性、可传递性、最优性和健壮性。这个项目汇集了一个由数学家、统计学家、理论计算机科学家和电气工程师组成的多学科团队，以发展深度学习的数学和科学基础。该项目分为四个主要部分。分析推力将利用逼近理论、信息论、统计推理和鲁棒控制等原理来分析深层网络的表现性、可解释性、置信度、公平性和稳健性。学习推力将利用动力系统、非凸和随机优化、统计学习理论、自适应控制和高维统计的原理来设计和分析具有保证的收敛、最优性和泛化性质的学习算法。设计主旨将使用代数、几何、拓扑、图论和优化的原理来设计和学习网络架构，这些架构在数据和任务中都捕获了代数、几何和图形结构。可转移性推力将使用多尺度分析和建模、强化学习和马尔可夫决策过程的原理来设计和研究适合于从多个任务中学习和转移到多个任务的数据表示法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Recent advances in deep learning have led to many disruptive technologies: from automatic speech recognition systems, to automated supermarkets, to self-driving cars. However, the complex and large-scale nature of deep networks makes them hard to analyze and, therefore, they are mostly used as black-boxes without formal guarantees on their performance. For example, deep networks provide a self-reported confidence score, but they are frequently inaccurate and uncalibrated, or likely to make large mistakes on rare cases. Moreover, the design of deep networks remains an art and is largely driven by empirical performance on a dataset. As deep learning systems are increasingly employed in our daily lives, it becomes critical to understand if their predictions satisfy certain desired properties. The goal of this NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning is to develop a mathematical, statistical and computational framework that helps explain the success of current network architectures, understand its pitfalls, and guide the design of novel architectures with guaranteed confidence, robustness, interpretability, optimality, and transferability. This project will train a diverse STEM workforce with data science skills that are essential for the global competitiveness of the US economy by creating new undergraduate and graduate programs in the foundations of data science and organizing a series of collaborative research events, including semester research programs and summer schools on the foundations of deep learning. This project will also impact women and underrepresented minorities by involving undergraduates in the foundations of data science.Deep networks have led to dramatic improvements in the performance of pattern recognition systems. However, the mathematical reasons for this success remain elusive. For instance, it is not clear why deep networks generalize or transfer to new tasks, or why simple optimization strategies can reach a local or global minimum of the associated non-convex optimization problem. Moreover, there is no principled way of designing the architecture of the network so that it satisfies certain desired properties, such as expressivity, transferability, optimality and robustness. This project brings together a multidisciplinary team of mathematicians, statisticians, theoretical computer scientists, and electrical engineers to develop the mathematical and scientific foundations of deep learning. The project is divided in four main thrusts. The analysis thrust will use principles from approximation theory, information theory, statistical inference, and robust control to analyze properties of deep networks such as expressivity, interpretability, confidence, fairness and robustness. The learning thrust will use principles from dynamical systems, non-convex and stochastic optimization, statistical learning theory, adaptive control, and high-dimensional statistics to design and analyze learning algorithms with guaranteed convergence, optimality and generalization properties. The design thrust will use principles from algebra, geometry, topology, graph theory and optimization to design and learn network architectures that capture algebraic, geometric and graph structures in both the data and the task. The transferability thrust will use principles from multiscale analysis and modeling, reinforcement learning, and Markov decision processes to design and study data representations that are suitable for learning from and transferring to multiple tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.