Building High-Throughput Neural Architecture Search Workflows via a Decoupled Fitness Prediction Engine
Building High-Throughput Neural Architecture Search Workflows via a Decoupled Fitness Prediction Engine
复制标题
DOI：
10.1109/tpds.2022.3140681
复制
发表时间：
2022-11-01
影响因子：
5.3
通讯作者：
Taufer, Michela
中科院分区：
文献类型：
--
作者：
Keller Rorabaugh, Ariel;Caino-Lores, Silvina;Taufer, Michela
神经网络（NN）用于高性能计算和高吞吐量分析，以从数据集中提取知识。神经架构搜索 (NAS) 通过生成、训练和分析数千个神经网络来自动化神经网络设计。然而，NAS 需要大量计算能力来进行神经网络训练。为了解决效率和可扩展性方面的挑战，我们提出了 PENGUIN，这是一种解耦的健身预测引擎，可以在不干扰搜索的情况下通知搜索。 PENGUIN 使用参数建模来预测神经网络的适应度。现有的 NAS 方法和参数化建模功能可以插入 PENGUIN 中，以构建灵活的 NAS 工作流程。通过这种解耦和灵活的参数化建模，PENGUIN 降低了训练成本：它预测 NN 的适应度，使 NAS 能够提前终止训练 NN。提前终止增加了固定计算资源可以评估的神经网络的数量，从而为 NAS 提供了寻找更好的神经网络的更多机会。我们使用 Summit 超级计算机在三个不同的基准数据集和三个最先进的 NAS 实现上评估了我们的引擎在 6,000 个神经网络上的有效性。使用 PENGUIN 增强这些 NAS 实施可以将吞吐量提高 1.6 至 7.1 倍。此外，walltime 测试表明 PENGUIN 可以将训练时间减少 2.5 至 5.3 倍。
Neural networks (NN) are used in high-performance computing and high-throughput analysis to extract knowledge from datasets. Neural architecture search (NAS) automates NN design by generating, training, and analyzing thousands of NNs. However, NAS requires massive computational power for NN training. To address challenges of efficiency and scalability, we propose PENGUIN, a decoupled fitness prediction engine that informs the search without interfering in it. PENGUIN uses parametric modeling to predict fitness of NNs. Existing NAS methods and parametric modeling functions can be plugged into PENGUIN to build flexible NAS workflows. Through this decoupling and flexible parametric modeling, PENGUIN reduces training costs: it predicts the fitness of NNs, enabling NAS to terminate training NNs early. Early termination increases the number of NNs that fixed compute resources can evaluate, thus giving NAS additional opportunity to find better NNs. We assess the effectiveness of our engine on 6,000 NNs across three diverse benchmark datasets and three state of the art NAS implementations using the Summit supercomputer. Augmenting these NAS implementations with PENGUIN can increase throughput by a factor of 1.6 to 7.1. Furthermore, walltime tests indicate that PENGUIN can reduce training time by a factor of 2.5 to 5.3.