CAREER: Optimization Landscape for Non-convex Functions - Towards Provable Algorithms for Neural Networks

深度学习是一种基于人工神经网络的机器学习方法，它大大提高了许多与理解自然图像、视频和语言等复杂数据相关的任务的学习算法的性能。基于深度学习的产品已经在人脸识别、机器翻译等领域产生了现实影响，并显示出在自动驾驶汽车等更多应用领域的前景。然而，尽管深度学习在实践中取得了成功，但对这些算法工作原理的理论理解却很少。理解深度学习算法的主要困难之一是，这些算法需要解决非常复杂的优化问题，试图找出神经元连接的最佳方式。在最一般的形式下，这些优化问题被认为是难以处理的。该研究项目将识别现实世界问题的特性，使这些问题具有特殊性和可处理性，并提供适用于深度学习的具有理论保证的新优化算法。项目中开发的材料将通过会议和研讨会传播，试图连接不同的研究社区，并用于创建新的机器学习课程。项目中设计的算法也将在标准深度学习框架中实现，并公开提供。该项目的具体方法围绕优化景观的新概念展开。对于一个优化问题，其优化景观包括对其局部和全局最优解的位置和值的清晰理解。研究目标分为三类。首先，研究项目将集中于一类局部可优化函数，其中局部最小值都是全局最优的。该研究项目将开发简单有效的算法来优化这些函数，以及一个新的框架来证明一些实际问题是局部可优化的。其次，该项目将开发更强大的优化算法，即使在优化环境不理想的情况下也能工作。最后，研究将集中在深度学习中出现的优化问题，并展示如何应用前两部分开发的技术。这些项目将为训练神经网络的启发式带来更多的理论见解。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning, a machine learning method that is based on artificial neural networks, has greatly improved the performance of learning algorithms for many tasks that are related to understanding complicated data such as natural images, videos and language. Products based on deep learning have already made real-life impact in face recognition, machine translation, and shown promise for more applications such as self-driving cars. However, despite the practical success of deep learning, theoretical understanding for why these algorithms work has been scarce. One of the main difficulties in understanding deep learning algorithms is that these algorithms need to solve very complicated optimization problems that try to find out what are the best ways for the neurons to be connected. In the most general form, these optimization problems are known to be intractable. This research project will identify properties of the real-world problems that make these problems special and tractable, and provide new optimization algorithms with theoretical guarantees that are applicable to deep learning. The materials developed in the project will be disseminated through conferences and workshops that try to connect different research communities, and used to create new machine learning courses. The algorithms designed in the project will also be implemented in standard deep learning frameworks and made publicly available.The specific approach of this project revolves around the new concept of optimization landscape. For an optimization problem, its optimization landscape includes clear understanding of the location and values of its local and global optimal solutions. The research goals are divided into three categories. First, the research project will focus on a class of locally optimizable functions for which local minima are all globally optimal. The research project will develop simple and efficient algorithms for optimizing such functions, as well as a new framework to prove several problems of practical interest are locally optimizable. Second, the project will develop stronger optimization algorithms that can work even when the optimization landscape is not as ideal. Finally, the research will focus on optimization problems that arise in deep learning and show how the techniques developed in the previous two parts can be applied. These projects will bring more theoretical insights into the heuristics for training neural networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.