机器学习问题的动量优化方法研究

无论是经典的正则化损失函数框架还是目前的研究热点深度学习，优化问题的高效求解都是一个不可或缺的关键环节。为了进一步改善梯度下降方法的性能，人们受物理学动量原理的启发开始使用动量优化方法。..动量方法的里程碑事件是使求解凸光滑问题梯度下降法的收敛速率获得了数量级式的加速。近期的研究表明动量方法可以提高深度学习的性能并指导网络的结构设计。但到目前为止，动量方法的研究仍然存在着一些欠缺，特别是非光滑问题很少涉及，不同动量方法的加速性和普适性也没有充分地体现。..本项目主要研究动量方法对一阶梯度优化算法收敛速率的加速问题和探索在深度学习和基于学习的优化中的应用。期待将动量技巧作为一种普适方法提升已有优化算法的性能，回答关于动量方法的争议，同时从动量方法角度给出Shamir在2012年COLT提出的强凸目标函数收敛性open问题另一种回答，并进一步期待动量优化方法可以克服深度学习的一些固有缺陷。

No matter in the classical regularized loss framework or in the state-of-the-art deep learning, it is an indispensable key part to solve the induced optimization problems efficiently. To further improve the performance of SGD, momentum-based optimization methods inspired by the physics principle have been used...The milestone about momentum methods is that it can accelerate the convergence rate of gradient descent methods by an order of magnitude when dealing with convex and smooth problems. Its recent successful application in deep learning reveals that momentum methods can remarkably improve performance of the deep neural networks and guide designation of their structures. However so far, there still exist some deficiencies. Especially, the nonsmooth optimization problems are seldom concerned and the acceleration and universality of different momentum methods are not sufficiently revealed...In this project, we mainly focus on investigating the role of momentum methods in acceleration of convergence rates of first-order gradient algorithms. Besides, their application in learning-based optimization as well as deep learning will be explored. As a result, we expect momentum technique can be used as a general method to improve the convergence performance of existing optimization algorithms, and some disputes about momentum methods can be answered. Simultaneously, we attempt to give another answer to the open problem from the perspective of momentum methods, which is about optimal convergence of the strongly-convex functions posed by Shamir in COLT2012. Furthermore, we also expect that the momentum methods can overcome some inherent defects in deep learning.