Auto-Tune: Structural Optimization of Machine Learning Frameworks for Large Datasets

我们的目标是使机器学习算法的设计自动化，以便促进非专家和自主系统的使用。尽管自动适应是机器学习的核心思想，但大多数算法仍然需要专家选择外部设计参数，这限制了它们的商业成功。我们的方法将好的算法配置的搜索形式化为不同机器学习算法的组合空间上的优化问题，并开发了新的贝叶斯优化算法来解决该问题。该项目遵循我们最近的Auto-WEKA框架的步骤，该框架证明了现代贝叶斯优化方法可以为非专家提供一种自动化(尽管计算非常昂贵)的方法来识别最新的复杂学习框架的实例化。下一步是在现实的预算约束下使这种方法可行，对于现代(大数据)和学习框架(特别是深度学习)，这通常意味着我们不能评估超过几个完整的模型实例。我们从人类实践者攻击新的学习问题的方式中获得灵感：将数据集与以前遇到的那些进行比较，并在数据的子集上评估一些有前途的方法，然后在整个数据集上只构建一个或几个模型。我们计划将所有这些组件集成到一个概率模型中，使用我们最近在涵盖这些维度的设计空间上进行熵搜索的贝叶斯优化算法，自动推导出类似于人类专家的设计策略。我们将通过改进现有的Auto-WEKA系统来验证我们的方法，并通过按下按钮实现第一种为新数据集学习有效深度网络的方法。我们提出了两个理论研究项目：1)算法性能的通用概率模型这一线索涉及寻找结构模型，该模型捕捉机器学习算法的高度结构化的相互依赖关系。2)预算节约型超参数优化以成本降低的形式运行机器学习算法通常是可行的，要么通过细化数据集，要么通过“关闭”算法的某些部分。我们的目标是将这种可能性编码到一种有成本意识的优化算法中，然后该算法应该能够自动控制从粗略原型到精细调整的过程。这两个理论进步将使两个应用项目成为可能：1)自动机器学习2)计算机视觉中的自动结构优化，特别是深度学习

We aim to automate the design of machine learning algorithms, in order to facilitate their use by non-experts and in autonomous systems. Although automated adaptation is a core idea of machine learning, most algorithms still require a choice of external design parameters by an expert, which limits their commercial success. Our approach formalizes the search for good algorithm configurations as an optimization problem over the combined space of different machine learning algorithms and develops novel Bayesian optimization algorithms for its solution.This projects follows in the steps of our recent Auto-WEKA framework, which demonstrated that modern Bayesian optimization methods can provide non-experts with an automated (albeit computationally very expensive) method to identify state-of-the-art instantiations of complex learning frameworks. The next step is to make this approach feasible under realistic budget constraints, which, for modern-day (big) datasets and learning frameworks (especially deep learning) often imply that we cannot evaluate more than a few full model instantiations.We take inspiration from the way human practitioners attack a new learning problem: compare the dataset to those previously encountered, and evaluate some promising methods on subsets of the data, before then only constructing one or a few models on the full dataset.We plan to integrate all these components into a probabilistic model, using our recent Bayesian optimization algorithm of Entropy Search on a design space covering these dimensions to automatically derive a strategy that resembles the design strategy of a human expert. We will validate our approaches by improving upon the existing Auto-WEKA system, and by implementing a first approach for learning an effective deep network for a new dataset at the push of a button.We propose two theoretical research projects: 1) General Probabilistic Models of Algorithm PerformanceThis thread involves finding structured models that capture the highly structured interdependences across the often very high-dimensional parameter spaces of machine learning algorithms.2) Budget-Thrifty Hyperparameter OptimizationIt is often feasible to run machine learning algorithms in a cost-reduced form, either by thinning the dataset or by "switching off" certain parts of an algorithm. We aim to encode this possibility in a cost-aware optimization algorithm, which should then be able to automatically control the progression from rough prototyping to fine tuning.These two theoretical advances will enable two applied projects, whichare: 1) Automatic Machine Learning2) Automated structural optimization in computer vision, especially deep learning