Collaborative Research: Transferable, Hierarchical, Expressive, Optimal, Robust, Interpretable Networks

深度学习的最新进展带来了许多颠覆性技术：从自动语音识别系统到自动超市，再到自动驾驶汽车。然而，深度网络的复杂性和大规模性使其难以分析，因此它们大多被用作黑箱，没有对其性能的正式保证。例如，深度网络提供了一个自我报告的信心评分，但它们通常不准确且未经校准，或者可能在极少数情况下犯下重大错误。此外，深度网络的设计仍然是一门艺术，并且在很大程度上由数据集上的经验表现驱动。随着深度学习系统越来越多地应用于我们的日常生活中，了解它们的预测是否满足某些期望的属性变得至关重要。NSF-Simons深度学习数学和科学基础研究合作的目标是开发一个数学，统计和计算框架，帮助解释当前网络架构的成功，了解其陷阱，并指导设计具有保证信心，鲁棒性，可解释性，最优性和可移植性的新型架构。该项目将通过在数据科学基础上创建新的本科和研究生课程，并组织一系列合作研究活动，包括学期研究计划和暑期学校，培养具有数据科学技能的多元化STEM劳动力，这些技能对美国经济的全球竞争力至关重要。该项目还将通过让本科生参与数据科学的基础来影响女性和代表性不足的少数群体。深度网络已经导致模式识别系统性能的显着改善。然而，这种成功的数学原因仍然难以捉摸。例如，目前尚不清楚为什么深度网络会泛化或转移到新任务，或者为什么简单的优化策略可以达到相关非凸优化问题的局部或全局最小值。此外，没有原则性的方法来设计网络的架构，使其满足某些期望的属性，如表达性，可转移性，最优性和鲁棒性。该项目汇集了数学家，统计学家，理论计算机科学家和电气工程师的多学科团队，以开发深度学习的数学和科学基础。该项目分为四个主要目标。分析重点将使用近似理论，信息论，统计推断和鲁棒控制的原理来分析深度网络的属性，如表达性，可解释性，置信度，公平性和鲁棒性。学习推力将使用动力系统，非凸和随机优化，统计学习理论，自适应控制和高维统计的原则，设计和分析学习算法，保证收敛，最优性和泛化性能。设计重点将使用代数，几何，拓扑，图论和优化的原理来设计和学习网络架构，以捕获数据和任务中的代数，几何和图形结构。可转移性推力将使用多尺度分析和建模、强化学习和马尔可夫决策过程的原理，设计和研究适合从多个任务中学习和转移到多个任务的数据表示。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。

Recent advances in deep learning have led to many disruptive technologies: from automatic speech recognition systems, to automated supermarkets, to self-driving cars. However, the complex and large-scale nature of deep networks makes them hard to analyze and, therefore, they are mostly used as black-boxes without formal guarantees on their performance. For example, deep networks provide a self-reported confidence score, but they are frequently inaccurate and uncalibrated, or likely to make large mistakes on rare cases. Moreover, the design of deep networks remains an art and is largely driven by empirical performance on a dataset. As deep learning systems are increasingly employed in our daily lives, it becomes critical to understand if their predictions satisfy certain desired properties. The goal of this NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning is to develop a mathematical, statistical and computational framework that helps explain the success of current network architectures, understand its pitfalls, and guide the design of novel architectures with guaranteed confidence, robustness, interpretability, optimality, and transferability. This project will train a diverse STEM workforce with data science skills that are essential for the global competitiveness of the US economy by creating new undergraduate and graduate programs in the foundations of data science and organizing a series of collaborative research events, including semester research programs and summer schools on the foundations of deep learning. This project will also impact women and underrepresented minorities by involving undergraduates in the foundations of data science.Deep networks have led to dramatic improvements in the performance of pattern recognition systems. However, the mathematical reasons for this success remain elusive. For instance, it is not clear why deep networks generalize or transfer to new tasks, or why simple optimization strategies can reach a local or global minimum of the associated non-convex optimization problem. Moreover, there is no principled way of designing the architecture of the network so that it satisfies certain desired properties, such as expressivity, transferability, optimality and robustness. This project brings together a multidisciplinary team of mathematicians, statisticians, theoretical computer scientists, and electrical engineers to develop the mathematical and scientific foundations of deep learning. The project is divided in four main thrusts. The analysis thrust will use principles from approximation theory, information theory, statistical inference, and robust control to analyze properties of deep networks such as expressivity, interpretability, confidence, fairness and robustness. The learning thrust will use principles from dynamical systems, non-convex and stochastic optimization, statistical learning theory, adaptive control, and high-dimensional statistics to design and analyze learning algorithms with guaranteed convergence, optimality and generalization properties. The design thrust will use principles from algebra, geometry, topology, graph theory and optimization to design and learn network architectures that capture algebraic, geometric and graph structures in both the data and the task. The transferability thrust will use principles from multiscale analysis and modeling, reinforcement learning, and Markov decision processes to design and study data representations that are suitable for learning from and transferring to multiple tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.