CIF: Small: A Probabilistic Theory of Deep Learning via Spline Operators

深度学习显著提高了解决各种困难的机器感知任务的能力，例如从图像中识别对象，从视频中识别活动，或从语音中识别文本。因此，深度学习系统不仅在从对话助手到无人驾驶汽车的新兴产品和服务中发挥着关键作用，而且还在从机器人技术到法律的文件分析等现有产品和服务中发挥着革命性作用。此外，在科学领域，深度学习正在为在大型复杂数据集中发现模式提供新的方法。这一成功令人印象深刻，但一个根本问题仍然存在：为什么深度学习有效？直觉比比皆是，但理解，分析和设计深度学习架构的连贯框架仍然难以捉摸。该项目将通过将深度学习系统与信号处理、近似理论、信息论和统计学的经典和最新结果联系起来，为深度学习系统奠定理论基础。一个关键目标是开发新型深度学习系统，其内部工作是可解释和可解释的。该项目将产生一系列影响，从为自主导航和决策等关键任务应用开发可信赖的、可解释的模型和算法，到推进机器学习和信号处理教育，该项目建立在基于分段仿射、凸非线性和最大仿射样条算子（MASO）的各类深度（神经）网络之间的优雅连接之上。这项研究是围绕两个相互关联的主题组织的。第一个主题围绕扩展的MASO框架超越分段仿射，凸非线性连接确定性MASO概率高斯混合模型。扩展的概率MASO将使深度网络的分析具有比分段仿射和凸网络（如sigmoid，双曲正切和softmax）更一般的非线性。第二个主题围绕将确定性MASO深度网络扩展到一类新的分层概率生成模型，这些模型通过封闭形式的变分期望最大化（EM）算法将传统深度网络的前馈推理计算和反向传播学习推广到最优贝叶斯推理。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning has significantly advanced the ability to address a wide range of difficult machine perception tasks, such as recognizing objects from images, activities from videos, or text from speech. As a result, deep learning systems not only are playing a key role in emerging products and services, from conversational assistants to driverless vehicles, but they are also revolutionizing existing ones, from robotics to legal document analysis. Moreover, in the scientific realm, deep learning is enabling new ways to find patterns in large complicated datasets. This success is impressive, but a fundamental question remains: Why does deep learning work? Intuitions abound, but a coherent framework for understanding, analyzing, and designing deep learning architectures has remained elusive. This project will develop a theoretical foundation for deep learning systems by connecting them to classical and recent results from the signal processing, approximation theory, information theory, and statistics. A key goal is the development of new kinds of deep learning systems whose inner workings are explainable and interpretable. This project will have a range of impacts, from developing trustworthy, interpretable models and algorithms for mission-critical applications like autonomous navigation and decision making to advancing machine learning and signal processing education.This project builds on an elegant connection between a wide class of deep (neural) networks based on piecewise-affine, convex nonlinearities and max-affine spline operators (MASOs). The research is organized around two interlocking themes. The first theme revolves around the extension of the MASO framework beyond piecewise-affine, convex nonlinearities by linking deterministic MASOs with probabilistic Gaussian mixture models. The extended, probabilistic MASO will enable the analysis of deep networks with more general nonlinearities than those that are piecewise-affine and convex, such as the sigmoid, hyperbolic tangent, and softmax. The second theme revolves around extending deterministic MASO deep networks to a new class of hierarchical, probabilistic, generative models that generalize the feedforward inference calculations and backpropagation learning of conventional deep networks to optimal Bayesian inference via a closed-form variational expectation-maximization (EM) algorithm. The probabilistic structure will enable the full arsenal of probability and statistics methodology to be applied to deep learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.