ADMM-SOFTMAX: AN ADMM APPROACH FOR MULTINOMIAL LOGISTIC REGRESSION
ADMM-SOFTMAX: AN ADMM APPROACH FOR MULTINOMIAL LOGISTIC REGRESSION
复制标题
DOI：
10.1553/etna_vol52s214
复制
发表时间：
2020-01-01
影响因子：
1.3
通讯作者：
Haber, Eldad
中科院分区：
文献类型：
--
作者：
Fung, Samy Wu;Tyrvainen, Sanna;Haber, Eldad
提出了一种求解多项逻辑回归问题的交替方向乘法器ADMM-Softmax。我们的方法适用于具有许多示例和特征的监督分类任务。该算法将MLR中的非线性优化问题简化为三步求解，从而有效地解决了MLR中的非线性优化问题。特别是，ADMM-Softmax的每次迭代都由一个线性最小二乘问题、一组独立的小规模光滑凸问题和一个平凡的对偶变量更新组成。最小二乘问题的解决方案可以通过预计算因子分解或预条件子来加速，并且光滑凸问题中的可分性可以很容易地在示例中并行化。对于两个图像分类问题，我们证明了ADMM-Softmax导致改进的推广相比，牛顿-克雷洛夫，拟牛顿，和随机梯度下降法。
We present ADMM-Softmax, an alternating direction method of multipliers (ADMM) for solving multinomial logistic regression (MLR) problems. Our method is geared toward supervised classification tasks with many examples and features. It decouples the nonlinear optimization problem in MLR into three steps that can be solved efficiently. In particular, each iteration of ADMM-Softmax consists of a linear least-squares problem, a set of independent small-scale smooth, convex problems, and a trivial dual variable update. The solution of the least-squares problem can be accelerated by pre-computing a factorization or preconditioner, and the separability in the smooth, convex problem can be easily parallelized across examples. For two image classification problems, we demonstrate that ADMM-Softmax leads to improved generalization compared to a Newton-Krylov, a quasi Newton, and a stochastic gradient descent method.