Minipatch Learning as Implicit Ridge-Like Regularization.
Minipatch Learning as Implicit Ridge-Like Regularization.
复制标题
DOI：
10.1109/bigcomp51126.2021.00021
复制
发表时间：
2021-01
期刊：
影响因子：
--
通讯作者：
Allen GI
中科院分区：
文献类型：
--
作者：
Yao T;LeJeune D;Javadi H;Baraniuk RG;Allen GI
脊状正则化通常通过减轻过拟合来提高机器学习模型的泛化性能。虽然脊正则化机器学习方法在许多重要应用中得到了广泛应用，但在具有数百万示例和特征的庞大数据场景中，通过优化进行直接训练可能会变得具有挑战性。我们提出了一种通用的方法，通过名为Minipatch Ridge（MPRigge）的隐式技术实现脊状正则化来解决这些挑战。我们的方法是基于对训练数据的示例和特征的许多微小的随机子样本进行训练的非正则化学习器的系数的集合，我们称之为minipatches。我们的经验表明，MPRigge诱导一个隐式的脊状正则化效果，并执行几乎相同的显式岭正则化的一般类别的预测，包括逻辑回归，SVM和鲁棒回归。令人尴尬的是，MPRigge提供了一种在计算上有吸引力的替代方法，以诱导脊状正则化，从而在具有挑战性的大数据设置中提高泛化性能。
Ridge-like regularization often leads to improved generalization performance of machine learning models by mitigating overfitting. While ridge-regularized machine learning methods are widely used in many important applications, direct training via optimization could become challenging in huge data scenarios with millions of examples and features. We tackle such challenges by proposing a general approach that achieves ridge-like regularization through implicit techniques named Minipatch Ridge (MPRidge). Our approach is based on taking an ensemble of coefficients of unregularized learners trained on many tiny, random subsamples of both the examples and features of the training data, which we call minipatches. We empirically demonstrate that MPRidge induces an implicit ridge-like regularizing effect and performs nearly the same as explicit ridge regularization for a general class of predictors including logistic regression, SVM, and robust regression. Embarrassingly parallelizable, MPRidge provides a computationally appealing alternative to inducing ridge-like regularization for improving generalization performance in challenging big-data settings.
影响因子：
25
作者：
Mostafavi S;Gaiteri C;Sullivan SE;White CC;Tasaki S;Xu J;Taga M;Klein HU;Patrick E;Komashko V;McCabe C;Smith R;Bradshaw EM;Root DE;Regev A;Yu L;Chibnik LB;Schneider JA;Young-Pearse TL;Bennett DA;De Jager PL
通讯作者：
De Jager PL
影响因子：
7.5
作者：
Breiman, L
通讯作者：
Breiman, L
影响因子：
1.1
作者：
Freund, Y;Schapire, RE
通讯作者：
Schapire, RE