Fast hyperparameter tuning using Bayesian optimization with directional derivatives
Fast hyperparameter tuning using Bayesian optimization with directional derivatives
复制标题
DOI：
10.1016/j.knosys.2020.106247
复制
发表时间：
2020-10-12
影响因子：
8.8
通讯作者：
Venkatesh, Svetha
中科院分区：
文献类型：
--
作者：
Joy, Tinu Theckel;Rana, Santu;Venkatesh, Svetha
受统计学习理论的启发，本文提出了一种基于贝叶斯优化的分类器超参数调整框架。我们利用了PAC学习理论中的两个关键事实：对于较小的数据子集，泛化范围将比整体更高；对于较小的数据子集，使用简单的模型可以达到最高的精度。我们最初使用贝叶斯优化在一小部分训练数据上调整超参数。在调整整个训练数据的超参数的同时，我们利用学习理论的见解来寻找更复杂的模型。我们通过在超参数搜索空间中策略性地放置方向导数符号来寻找比用小数据获得的模型更复杂的模型来实现这一点。我们在几种机器学习算法的超参数调整任务中展示了我们的方法的性能。(C)2020爱思唯尔B.V.保留所有权利。
In this paper we develop a Bayesian optimization based hyperparameter tuning framework inspired by statistical learning theory for classifiers. We utilize two key facts from PAC learning theory; the generalization bound will be higher for a small subset of data compared to the whole, and the highest accuracy for a small subset of data can be achieved with a simple model. We initially tune the hyperparameters on a small subset of training data using Bayesian optimization. While tuning the hyperparameters on the whole training data, we leverage the insights from the learning theory to seek more complex models. We realize this by using directional derivative signs strategically placed in the hyperparameter search space to seek a more complex model than the one obtained with small data. We demonstrate the performance of our method on the tasks of tuning the hyperparameters of several machine learning algorithms. (C) 2020 Elsevier B.V. All rights reserved.