SHADHO: Massively Scalable Hardware-Aware Distributed Hyperparameter Optimization
SHADHO: Massively Scalable Hardware-Aware Distributed Hyperparameter Optimization
复制标题
DOI：
10.1109/wacv.2018.00086
复制
发表时间：
2017-07
期刊：
影响因子：
--
通讯作者：
Jeffery Kinnison;Nathaniel Kremer-Herman;D. Thain;W. Scheirer
中科院分区：
文献类型：
--
作者：
Jeffery Kinnison;Nathaniel Kremer-Herman;D. Thain;W. Scheirer
计算机视觉正在经历人工智能的复兴，其中机器学习模型正在加速学术研究和商业应用的重要突破。然而，有效地训练这些模型并非易事，部分原因在于超参数：用户配置的值控制模型从数据中学习的能力。现有的超参数优化方法是高度并行的，但没有努力平衡跨异构硬件的搜索或优先搜索高影响空间。本文介绍了一种大规模可扩展硬件感知分布式超参数优化（SHADHO）框架。我们的框架计算每个搜索空间的相对复杂性，并在所有试验中监控学习任务的性能。然后将这些指标用作启发式方法，根据分布式工作人员的硬件为其分配超参数。我们首先证明，我们的框架通过使用150个分布式工作者优化MNIST的SVM，实现了标准分布式超参数优化框架的两倍吞吐量。然后，我们在一周的时间里使用SHADHO进行模型搜索，在两个计算集群上使用74个gpu来优化U-Net的细胞分割任务，发现515个模型实现了比标准U-Net更低的验证损失。
Computer vision is experiencing an AI renaissance, in which machine learning models are expediting important breakthroughs in academic research and commercial applications. Effectively training these models, however, is not trivial due in part to hyperparameters: user-configured values that control a model's ability to learn from data. Existing hyperparameter optimization methods are highly parallel but make no effort to balance the search across heterogeneous hardware or to prioritize searching high-impact spaces. In this paper, we introduce a framework for massively Scalable Hardware-Aware Distributed Hyperparameter Optimization (SHADHO). Our framework calculates the relative complexity of each search space and monitors performance on the learning task over all trials. These metrics are then used as heuristics to assign hyperparameters to distributed workers based on their hardware. We first demonstrate that our framework achieves double the throughput of a standard distributed hyperparameter optimization framework by optimizing SVM for MNIST using 150 distributed workers. We then conduct model search with SHADHO over the course of one week using 74 GPUs across two compute clusters to optimize U-Net for a cell segmentation task, discovering 515 models that achieve a lower validation loss than standard U-Net.