HY-POP: Hyperparameter optimization of machine learning models through parametric programming
HY-POP: Hyperparameter optimization of machine learning models through parametric programming
复制标题
DOI：
10.1016/j.compchemeng.2020.106902
复制
发表时间：
2020-08
期刊：
影响因子：
--
通讯作者：
William W. Tso;B. Burnak;E. Pistikopoulos
中科院分区：
文献类型：
--
作者：
William W. Tso;B. Burnak;E. Pistikopoulos
拟合机器学习模型通常需要预先设置参数值(超参数)，以控制算法如何从数据中学习。选择误差最小且适用于未知数据的最优模型成为调整或优化这些超参数的问题。典型的超参数优化策略包括对参数空间进行离散化，并通过交叉验证实现迭代搜索过程来逼近最优超参数和模型选择。取而代之的是，对于线性或二次规划(LP/QP)模型的机器学习算法，可以通过参数规划获得超参数优化问题的精确解，而不需要任何近似。首先，超参数优化问题更自然地被提出为一个双层优化问题。其次，利用参数规划理论，将双层优化问题转化为单层优化问题。推导了套索回归和LPL1范数支持向量机的超参数优化问题的精确解，并用实例数据进行了验证。
Fitting a machine learning model often requires presetting parameter values (hyperparameters) that control how an algorithm learns from the data. Selecting an optimal model that minimizes error and generalizes well to unseen data becomes a problem of tuning or optimizing these hyperparameters. Typical hyperparameter optimization strategies involve discretizing the parameter space and implementing an iterative search procedure to approximate the optimal hyperparameter and model selection through cross-validation. Instead, for machine learning algorithms that are formulated as linear or quadratic programming (LP/QP) models, an exact solution to the hyperparameter optimization problem is obtainable through parametric programming without any approximation. First, the hyperparameter optimization problem is posed more naturally as a bilevel optimization. Second, using parametric programming theory, the bilevel optimization is reformulated into a single level problem. Exact solutions to the hyperparameter optimization problem for LASSO regression and LPL1-norm support vector machine (SVM) are derived and validated on example data.