RI: Small: An Optimization Framework for Understanding Deep Networks

在过去的几年里，由于引入了深度神经网络，模式识别系统的性能有了显著的提高。然而，这种成功的数学原因仍然难以捉摸。一个关键的挑战是神经网络参数的学习问题是一个非凸优化问题，这使得找到全局最优参数非常困难。另一个挑战是，目前关于如何构建网络架构的理论非常有限（即，层数，每层神经元的数量，连接模式等）。该项目的目标是开发一个优化框架，为当前网络架构的成功提供理论见解，并指导设计具有全局最优性的新架构。该项目将开发一个数学框架，用于分析一类广泛的非凸优化问题，包括矩阵分解、张量分解和深度学习。特别地，本项目将研究损失函数和正则函数的和的最小化问题，这两个函数都可以是非凸的，但必须满足一定的“正齐次性”。通过适当地设计约束“网络大小”的正齐次正则化器，该项目旨在表明，在某些条件下，所有局部最小值都是全局最优的，并且可以使用局部下降策略从任何初始化中找到全局最小值。深入了解深度网络的数学特性不仅会影响机器学习和优化，因为我们对非凸问题的理解仍然非常有限，而且还会影响计算机视觉、语音和自然语言处理等应用领域，深度网络目前在这些领域提供了最先进的结果。

The past few years have seen a dramatic increase in the performance of pattern recognition systems due to the introduction of deep neural networks. However, the mathematical reasons for this success remain elusive. A key challenge is that the problem of learning the parameters of a neural network is a non-convex optimization problem, which makes finding the globally optimal parameters extremely difficult. Another challenge is that there is currently very limited theory about how the network architecture should be constructed (i.e., number of layers, number of neurons per layer, connectivity patterns, etc.). The goal of this project is to develop an optimization framework that provides theoretical insights for the success of current network architectures and guides the design of novel architectures with guarantees of global optimality. This project will develop a mathematical framework for the analysis of a broad class of non-convex optimization problems, including matrix factorization, tensor factorization, and deep learning. In particular, this project will study the problem of minimizing the sum of a loss function and a regularization function, both of which can be non-convex, but should satisfy a certain "positive homogeneity" property. By properly designing positively homogeneous regularizers that constrain the "network size," this project aims to show that, under certain conditions, all local minima are globally optimal, and one can find a global minimum from any initialization using a local descent strategy. A deeper understanding of the mathematical properties of deep networks will impact not only machine learning and optimization, where our understanding of non-convex problems continues to be very limited, but also application areas such as computer vision, speech and natural language processing, where deep networks currently give state-of-the-art results.