Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizers
Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizers
复制标题
DOI：
10.1109/tevc.2022.3211336
复制
发表时间：
2021-11
影响因子：
14.3
通讯作者：
Julia Moosbauer;Martin Binder;Lennart Schneider;Florian Pfisterer;Marc Becker;Michel Lang;Lars Kotthoff;Bernd Bischl
中科院分区：
文献类型：
--
作者：
Julia Moosbauer;Martin Binder;Lennart Schneider;Florian Pfisterer;Marc Becker;Michel Lang;Lars Kotthoff;Bernd Bischl
自动化超参数优化（HPO）已经得到了广泛的应用，是大多数自动化机器学习框架的重要组成部分。然而，设计HPO算法的过程仍然是一个非系统的和手动的过程：新算法通常是在先前工作的基础上建立的，其中发现了局限性并提出了改进建议。尽管这种方法是由专家知识指导的，但它仍然有些武断。这个过程很少允许获得对哪些算法组件驱动性能的整体理解，并且有忽视良好算法设计选择的风险。我们提出了一种应用于多保真HPO （MF-HPO）的自动基准驱动算法设计的原则方法。首先，我们形式化了一个丰富的MF-HPO候选空间，其中包括但不限于常见的现有HPO算法，然后提出了一个覆盖该空间的可配置框架。为了自动和系统地找到最佳候选，我们采用了一种基于优化的编程方法，并通过贝叶斯优化在算法候选空间中进行搜索。我们通过执行消融分析来质疑所发现的设计选择是必要的还是可以被更幼稚和更简单的设计选择所取代。我们观察到，只要将一些关键配置参数设置为正确的值，使用相对简单的配置（在某些方面，比已建立的方法更简单）就可以执行得非常好。
Automated hyperparameter optimization (HPO) has gained great popularity and is an important component of most automated machine learning frameworks. However, the process of designing HPO algorithms is still an unsystematic and manual process: new algorithms are often built on top of prior work, where limitations are identified and improvements are proposed. Even though this approach is guided by expert knowledge, it is still somewhat arbitrary. The process rarely allows for gaining a holistic understanding of which algorithmic components drive performance and carries the risk of overlooking good algorithmic design choices. We present a principled approach to automated benchmark-driven algorithm design applied to multifidelity HPO (MF-HPO). First, we formalize a rich space of MF-HPO candidates that includes, but is not limited to, common existing HPO algorithms and then present a configurable framework covering this space. To find the best candidate automatically and systematically, we follow a programming-by-optimization approach and search over the space of algorithm candidates via Bayesian optimization. We challenge whether the found design choices are necessary or could be replaced by more naive and simpler ones by performing an ablation analysis. We observe that using a relatively simple configuration (in some ways, simpler than established methods) performs very well as long as some critical configuration parameters are set to the right value.