Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach
Bayesian Optimization Over Iterative Learners with Structured Responses: A Budget-aware Planning Approach
复制标题
DOI：
10.48550/arxiv.2206.12708
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Syrine Belakaria;Rishit Sheth;J. Doppa;Nicoló Fusi
中科院分区：
文献类型：
--
作者：
Syrine Belakaria;Rishit Sheth;J. Doppa;Nicoló Fusi
深度神经网络 (DNN) 和数据集规模的不断增长激发了对同步模型选择和训练的有效解决方案的需求。许多迭代学习器的超参数优化 (HPO) 方法（包括 DNN）都试图通过查询和学习响应面并同时搜索响应面的最优值来解决此问题。然而，这些方法中的许多方法都会进行短视查询，不考虑有关响应结构的先验知识，和/或执行有偏差的成本感知搜索，所有这些都会在指定总成本预算时加剧识别最佳性能模型。本文提出了一种称为{\bf B}udget-{\bf A}ware {\bf P}lanning 的新颖方法，用于{\bf I}迭代学习器 (BAPI)，以在有限的成本预算下解决 HPO 问题。 BAPI 是一种高效的非近视贝叶斯优化解决方案，它考虑预算并利用有关目标函数和成本函数的先验知识来选择更好的配置并在评估（训练）期间做出更明智的决策。对迭代学习器的各种 HPO 基准进行的实验表明，在大多数情况下，BAPI 的性能优于最先进的基准。
The rising growth of deep neural networks (DNNs) and datasets in size motivates the need for efficient solutions for simultaneous model selection and training. Many methods for hyperparameter optimization (HPO) of iterative learners, including DNNs, attempt to solve this problem by querying and learning a response surface while searching for the optimum of that surface. However, many of these methods make myopic queries, do not consider prior knowledge about the response structure, and/or perform a biased cost-aware search, all of which exacerbate identifying the best-performing model when a total cost budget is specified. This paper proposes a novel approach referred to as {\bf B}udget-{\bf A}ware {\bf P}lanning for {\bf I}terative Learners (BAPI) to solve HPO problems under a constrained cost budget. BAPI is an efficient non-myopic Bayesian optimization solution that accounts for the budget and leverages the prior knowledge about the objective function and cost function to select better configurations and to take more informed decisions during the evaluation (training). Experiments on diverse HPO benchmarks for iterative learners show that BAPI performs better than state-of-the-art baselines in most cases.