Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks
Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks
复制标题
DOI：
--
复制
发表时间：
2021-02
期刊：
影响因子：
--
通讯作者：
Lemeng Wu;Bo Liu-;P. Stone;Qiang Liu
中科院分区：
文献类型：
--
作者：
Lemeng Wu;Bo Liu-;P. Stone;Qiang Liu
我们提出了萤火虫神经架构下降，渐进和动态增长的神经网络，以共同优化网络的参数和架构的一般框架。我们的方法以最速下降的方式工作，它迭代地在原始网络的功能邻域内找到最佳网络，其中包括一组不同的候选网络结构。通过泰勒近似，可以找到最佳的网络结构的邻域内的贪婪选择过程。我们表明，萤火虫的后裔可以灵活地使网络更宽更深，并且可以应用于学习准确但资源高效的神经架构，以避免在持续学习中发生灾难性遗忘。从经验上讲，萤火虫血统在神经结构搜索和持续学习方面都取得了可喜的成果。特别是，在一个具有挑战性的连续图像分类任务中，它学习的网络尺寸较小，但平均精度高于最先进的方法。
We propose firefly neural architecture descent, a general framework for progressively and dynamically growing neural networks to jointly optimize the networks' parameters and architectures. Our method works in a steepest descent fashion, which iteratively finds the best network within a functional neighborhood of the original network that includes a diverse set of candidate network structures. By using Taylor approximation, the optimal network structure in the neighborhood can be found with a greedy selection procedure. We show that firefly descent can flexibly grow networks both wider and deeper, and can be applied to learn accurate but resource-efficient neural architectures that avoid catastrophic forgetting in continual learning. Empirically, firefly descent achieves promising results on both neural architecture search and continual learning. In particular, on a challenging continual image classification task, it learns networks that are smaller in size but have higher average accuracy than those learned by the state-of-the-art methods.