Auto-Tune: Structural Optimization of Machine Learning Frameworks for Large Datasets

我们的目标是使机器学习算法的设计自动化，以便于非专家和自主系统中使用它们。尽管自动适应是机器学习的核心思想，但大多数算法仍然需要专家选择外部设计参数，这限制了它们的商业成功。我们的方法将寻找好的算法配置形式化为不同机器学习算法组合空间上的优化问题，并为其解决方案开发了新的贝叶斯优化算法。该项目遵循我们最近的Auto-WEKA框架的步骤，该框架证明了现代贝叶斯优化方法可以为非专家提供一种自动化（尽管计算上非常昂贵）的方法来识别复杂学习框架的最新实例。下一步是使这种方法在现实的预算限制下可行，这对于现代（大）数据集和学习框架（特别是深度学习）通常意味着我们无法评估多个完整的模型实例。我们从人类从业者解决新学习问题的方式中获得灵感：将数据集与之前遇到的数据集进行比较，并在数据子集上评估一些有前途的方法，然后在整个数据集上仅构建一个或几个模型。我们计划将所有这些组件集成到一个概率模型中，在覆盖这些维度的设计空间上使用我们最近的熵搜索贝叶斯优化算法来自动导出类似于人类专家设计策略的策略。我们将通过改进现有的Auto-WEKA系统来验证我们的方法，并通过实施第一种方法，在按下按钮时为新数据集学习有效的深度网络。我们提出了两个理论研究项目：1)算法性能的一般概率模型这个线程涉及寻找结构化模型，这些模型可以捕获机器学习算法中通常非常高维的参数空间中高度结构化的相互依赖性。2) Budget-Thrifty超参数优化以降低成本的形式运行机器学习算法通常是可行的，要么通过细化数据集，要么通过“关闭”算法的某些部分。我们的目标是在成本感知优化算法中编码这种可能性，然后它应该能够自动控制从粗糙原型到微调的进程。这两项理论进展将使两个应用项目成为可能：1)自动机器学习2)计算机视觉中的自动结构优化，特别是深度学习

We aim to automate the design of machine learning algorithms, in order to facilitate their use by non-experts and in autonomous systems. Although automated adaptation is a core idea of machine learning, most algorithms still require a choice of external design parameters by an expert, which limits their commercial success. Our approach formalizes the search for good algorithm configurations as an optimization problem over the combined space of different machine learning algorithms and develops novel Bayesian optimization algorithms for its solution.This projects follows in the steps of our recent Auto-WEKA framework, which demonstrated that modern Bayesian optimization methods can provide non-experts with an automated (albeit computationally very expensive) method to identify state-of-the-art instantiations of complex learning frameworks. The next step is to make this approach feasible under realistic budget constraints, which, for modern-day (big) datasets and learning frameworks (especially deep learning) often imply that we cannot evaluate more than a few full model instantiations.We take inspiration from the way human practitioners attack a new learning problem: compare the dataset to those previously encountered, and evaluate some promising methods on subsets of the data, before then only constructing one or a few models on the full dataset.We plan to integrate all these components into a probabilistic model, using our recent Bayesian optimization algorithm of Entropy Search on a design space covering these dimensions to automatically derive a strategy that resembles the design strategy of a human expert. We will validate our approaches by improving upon the existing Auto-WEKA system, and by implementing a first approach for learning an effective deep network for a new dataset at the push of a button.We propose two theoretical research projects: 1) General Probabilistic Models of Algorithm PerformanceThis thread involves finding structured models that capture the highly structured interdependences across the often very high-dimensional parameter spaces of machine learning algorithms.2) Budget-Thrifty Hyperparameter OptimizationIt is often feasible to run machine learning algorithms in a cost-reduced form, either by thinning the dataset or by "switching off" certain parts of an algorithm. We aim to encode this possibility in a cost-aware optimization algorithm, which should then be able to automatically control the progression from rough prototyping to fine tuning.These two theoretical advances will enable two applied projects, whichare: 1) Automatic Machine Learning2) Automated structural optimization in computer vision, especially deep learning