When will gradient methods converge to max‐margin classifier under ReLU models?
When will gradient methods converge to max‐margin classifier under ReLU models?
复制标题
DOI：
10.1002/sta4.354
复制
发表时间：
2018-06
期刊：
影响因子：
1.7
通讯作者：
Tengyu Xu;Yi Zhou;Kaiyi Ji;Yingbin Liang
中科院分区：
文献类型：
--
作者：
Tengyu Xu;Yi Zhou;Kaiyi Ji;Yingbin Liang
研究了线性可分数据集上梯度下降法求解二元分类问题时的隐式偏差。分类器由非线性ReLU模型描述，目标函数采用指数损失函数。我们首先描述景观的损失函数，并表明，可以存在虚假的渐近局部极小值，除了渐近全局极小值。然后，我们证明了梯度下降（GD）可以收敛到全局或局部最大边缘方向，或者在一般情况下可能偏离所需的最大边缘方向。对于随机梯度下降（SGD），我们证明了如果SGD收敛，则它在期望中收敛到全局或局部最大边缘方向。我们进一步探索了这些算法在某些静态条件下学习多神经元网络时的隐式偏差，并表明学习的分类器在ReLU激活下最大化了每个样本模式分区的边缘。
We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable data set. The classifier is described by a non‐linear ReLU model and the objective function adopts the exponential loss function. We first characterize the landscape of the loss function and show that there can exist spurious asymptotic local minima besides asymptotic global minima. We then show that gradient descent (GD) can converge to either a global or a local max‐margin direction or may diverge from the desired max‐margin direction in a general context. For stochastic gradient descent (SGD), we show that it converges in expectation to either the global or the local max‐margin direction if SGD converges. We further explore the implicit bias of these algorithms in learning a multineuron network under certain stationary conditions and show that the learned classifier maximizes the margins of each sample pattern partition under the ReLU activation.