RI: Small: An Optimization Framework for Understanding Deep Networks

RI：小型：理解深度网络的优化框架

• 批准号: 1618485
• 负责人: Rene Vidal
• 金额: $ 45万
• 依托单位: Johns Hopkins University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-07-01 至 2020-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1618485&HistoricalAwards=false
• 关键词: RI; Small; Optimization; Framework; Understanding

The past few years have seen a dramatic increase in the performance of pattern recognition systems due to the introduction of deep neural networks. However, the mathematical reasons for this success remain elusive. A key challenge is that the problem of learning the parameters of a neural network is a non-convex optimization problem, which makes finding the globally optimal parameters extremely difficult. Another challenge is that there is currently very limited theory about how the network architecture should be constructed (i.e., number of layers, number of neurons per layer, connectivity patterns, etc.). The goal of this project is to develop an optimization framework that provides theoretical insights for the success of current network architectures and guides the design of novel architectures with guarantees of global optimality. This project will develop a mathematical framework for the analysis of a broad class of non-convex optimization problems, including matrix factorization, tensor factorization, and deep learning. In particular, this project will study the problem of minimizing the sum of a loss function and a regularization function, both of which can be non-convex, but should satisfy a certain "positive homogeneity" property. By properly designing positively homogeneous regularizers that constrain the "network size," this project aims to show that, under certain conditions, all local minima are globally optimal, and one can find a global minimum from any initialization using a local descent strategy. A deeper understanding of the mathematical properties of deep networks will impact not only machine learning and optimization, where our understanding of non-convex problems continues to be very limited, but also application areas such as computer vision, speech and natural language processing, where deep networks currently give state-of-the-art results.

在过去的几年里，由于深度神经网络的引入，模式识别系统的性能大幅提高。然而，这种成功的数学原因仍然难以捉摸。一个关键的挑战是，学习神经网络的参数的问题是一个非凸优化问题，这使得寻找全局最优参数非常困难。另一个挑战是目前关于网络体系结构应该如何构造的理论非常有限（即，层数、每层神经元的数量、连接模式等）。该项目的目标是开发一个优化框架，为当前网络架构的成功提供理论见解，并指导新架构的设计，保证全局最优。该项目将开发一个数学框架，用于分析广泛的非凸优化问题，包括矩阵分解，张量分解和深度学习。特别是，这个项目将研究最小化损失函数和正则化函数之和的问题，这两个函数都可以是非凸的，但应该满足一定的“正齐性”性质。通过适当地设计约束“网络大小”的正齐次正则化器，该项目旨在表明，在某些条件下，所有局部最小值都是全局最优的，并且可以使用局部下降策略从任何初始化中找到全局最小值。对深度网络数学特性的深入理解不仅会影响机器学习和优化，我们对非凸问题的理解仍然非常有限，而且还会影响计算机视觉，语音和自然语言处理等应用领域，深度网络目前提供最先进的结果。