Developing Learning Algorithms via Optimized Discretization of Continuous Dynamical Systems
Developing Learning Algorithms via Optimized Discretization of Continuous Dynamical Systems
复制标题
DOI：
10.1109/tsmcb.2011.2163506
复制
发表时间：
2012-02
期刊：
影响因子：
--
通讯作者：
Qing Tao;Zhengya Sun;Kang-Kook Kong
中科院分区：
文献类型：
--
作者：
Qing Tao;Zhengya Sun;Kang-Kook Kong
现有的大多数数值优化方法都是基于一些常微分方程的离散化。为了解决机器学习中的一些凸优化和光滑优化问题，本文基于一个新的原理，即，连续动力系统的优化离散化（ODCDS）。首先，引入了一个具有李雅普诺夫稳定性和单调性的批量学习投影梯度动力系统，其动力学行为保证了基于离散化的优化器的精确性和线搜索策略的适用性。在公平的假设下，得到了一个新的在线学习算法，其后悔度为O（logT）或O（logT）.通过使用线搜索策略，建议的批量学习ODCDS表现出不敏感的步长和更快的下降。该算法只需少量的线搜索步，就具有足够的稳定性和近似最优性。实验结果证明了理论分析的正确性和算法的有效性。
Most of the existing numerical optimization methods are based upon a discretization of some ordinary differential equations. In order to solve some convex and smooth optimization problems coming from machine learning, in this paper, we develop efficient batch and online algorithms based on a new principle, i.e., the optimized discretization of continuous dynamical systems (ODCDSs). First, a batch learning projected gradient dynamical system with Lyapunov's stability and monotonic property is introduced, and its dynamical behavior guarantees the accuracy of discretization-based optimizer and applicability of line search strategy. Furthermore, under fair assumptions, a new online learning algorithm achieving regret O(√T) or O(logT) is obtained. By using the line search strategy, the proposed batch learning ODCDS exhibits insensitivity to the step sizes and faster decrease. With only a small number of line search steps, the proposed stochastic algorithm shows sufficient stability and approximate optimality. Experimental results demonstrate the correctness of our theoretical analysis and efficiency of our algorithms.