CIF: Small: A Probabilistic Theory of Deep Learning via Spline Operators

深度学习极大地提高了解决各种困难的机器感知任务的能力，例如从图像中识别物体，从视频中识别活动，或从语音中识别文本。因此，深度学习系统不仅在新兴产品和服务中发挥着关键作用，从会话助手到无人驾驶汽车，而且还在彻底改变现有的产品和服务，从机器人到法律文件分析。此外，在科学领域，深度学习为在大型复杂数据集中发现模式提供了新的方法。这种成功令人印象深刻，但一个基本问题仍然存在：为什么深度学习有效？直觉比比皆是，但理解、分析和设计深度学习架构的连贯框架仍然难以捉摸。该项目将通过将深度学习系统与信号处理、近似理论、信息论和统计学的经典和最新结果联系起来，为深度学习系统奠定理论基础。一个关键目标是开发新型的深度学习系统，其内部工作是可解释和可解释的。该项目将产生一系列影响，从为自主导航和决策等关键任务应用开发可信赖、可解释的模型和算法，到推进机器学习和信号处理教育。该项目建立在基于分段仿射、凸非线性和最大仿射样条算子（MASOs）的广泛的深度（神经）网络之间的优雅连接上。这项研究围绕两个相互关联的主题展开。第一个主题围绕着通过将确定性MASO与概率高斯混合模型联系起来，将MASO框架扩展到分段仿射、凸非线性之外。扩展的、概率的MASO将使深度网络的分析具有更一般的非线性，而不是那些分段仿射和凸的，如s型、双曲正切和softmax。第二个主题围绕着将确定性MASO深度网络扩展到一类新的分层、概率、生成模型，这些模型将传统深度网络的前馈推理计算和反向传播学习推广到最优贝叶斯推理，通过封闭形式的变分期望最大化（EM）算法。概率结构将使概率和统计方法的全部武器库应用于深度学习。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning has significantly advanced the ability to address a wide range of difficult machine perception tasks, such as recognizing objects from images, activities from videos, or text from speech. As a result, deep learning systems not only are playing a key role in emerging products and services, from conversational assistants to driverless vehicles, but they are also revolutionizing existing ones, from robotics to legal document analysis. Moreover, in the scientific realm, deep learning is enabling new ways to find patterns in large complicated datasets. This success is impressive, but a fundamental question remains: Why does deep learning work? Intuitions abound, but a coherent framework for understanding, analyzing, and designing deep learning architectures has remained elusive. This project will develop a theoretical foundation for deep learning systems by connecting them to classical and recent results from the signal processing, approximation theory, information theory, and statistics. A key goal is the development of new kinds of deep learning systems whose inner workings are explainable and interpretable. This project will have a range of impacts, from developing trustworthy, interpretable models and algorithms for mission-critical applications like autonomous navigation and decision making to advancing machine learning and signal processing education.This project builds on an elegant connection between a wide class of deep (neural) networks based on piecewise-affine, convex nonlinearities and max-affine spline operators (MASOs). The research is organized around two interlocking themes. The first theme revolves around the extension of the MASO framework beyond piecewise-affine, convex nonlinearities by linking deterministic MASOs with probabilistic Gaussian mixture models. The extended, probabilistic MASO will enable the analysis of deep networks with more general nonlinearities than those that are piecewise-affine and convex, such as the sigmoid, hyperbolic tangent, and softmax. The second theme revolves around extending deterministic MASO deep networks to a new class of hierarchical, probabilistic, generative models that generalize the feedforward inference calculations and backpropagation learning of conventional deep networks to optimal Bayesian inference via a closed-form variational expectation-maximization (EM) algorithm. The probabilistic structure will enable the full arsenal of probability and statistics methodology to be applied to deep learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.