Hyperparameter Learning via Bilevel Nonsmooth Optimization
Hyperparameter Learning via Bilevel Nonsmooth Optimization
复制标题
DOI：
--
复制
发表时间：
2018-06
期刊：
影响因子：
--
通讯作者：
Takayuki Okuno;A. Takeda;Akihiro Kawana
中科院分区：
文献类型：
--
作者：
Takayuki Okuno;A. Takeda;Akihiro Kawana
我们提出了一个双层优化策略，以选择最佳的超参数值的非光滑$\ell_p$正则化0 <p\le 1$。有关的双层优化问题有一个非光滑的，可能是非凸的，$\ell_p$-正则化的问题作为下层问题。尽管最近流行的非凸$\ell_p$正则化和有用的双层优化选择超参数，算法这样的双层问题还没有研究，因为困难的$\ell_p$正则化。我们首先显示新的最优性条件，这样的双层优化问题，然后提出了一个平滑型算法的收敛性分析。该算法是简单和可扩展的，我们的数值比较贝叶斯优化和网格搜索表明。作为第一个保证收敛的算法，它是求解非光滑非凸双层优化问题的一个很有前途的算法。
We propose a bilevel optimization strategy for selecting the best hyperparameter value for the nonsmooth $\ell_p$ regularizer with $0<p\le 1$. The concerned bilevel optimization problem has a nonsmooth, possibly nonconvex, $\ell_p$-regularized problem as the lower-level problem. Despite the recent popularity of nonconvex $\ell_p$ regularizer and the usefulness of bilevel optimization for selecting hyperparameters, algorithms for such bilevel problems have not been studied because of the difficulty of $\ell_p$ regularizer. We first show new optimality conditions for such bilevel optimization problems and then propose a smoothing-type algorithm together with convergence analysis. The proposed algorithm is simple and scalable as our numerical comparison to Bayesian optimization and grid search indicates. It is a promising algorithm for nonsmooth nonconvex bilevel optimization problems as the first algorithm with convergence guarantee.