Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
复制标题
DOI：
--
复制
发表时间：
2020-02
期刊：
影响因子：
--
通讯作者：
Xiangning Chen;Cho-Jui Hsieh
中科院分区：
文献类型：
--
作者：
Xiangning Chen;Cho-Jui Hsieh
可区分的体系结构搜索（DARTS）是用于识别体系结构的流行NAS解决方案。基于对体系结构空间的不断松弛，DARTS学习可微的体系结构权重，大大降低了搜索成本。然而，随着搜索的进行，它的稳定性受到了挑战，因为它产生了不断恶化的体系结构。我们发现，陡峭的验证损失景观，这导致了一个戏剧性的性能下降时，蒸馏的最终架构，是一个重要的因素，导致不稳定。基于这一观察，我们提出了一种基于扰动的正则化- SmoothDARTS（SDARTS），以平滑损失景观并提高基于DARTS的方法的泛化能力。特别是，我们的新配方通过随机平滑或对抗攻击来稳定基于DARTS的方法。NAS-Bench-1 Shot 1上的搜索轨迹证明了我们方法的有效性，由于稳定性的提高，我们在4个数据集上的各种搜索空间中实现了性能提升。此外，我们在数学上表明，SDARTS隐式地正则化了验证损失的Hessian范数，这导致了更平滑的损失景观和更好的性能。
Differentiable architecture search (DARTS) is a prevailing NAS solution to identify architectures. Based on the continuous relaxation of the architecture space, DARTS learns a differentiable architecture weight and largely reduces the search cost. However, its stability has been challenged for yielding deteriorating architectures as the search proceeds. We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. Based on this observation, we propose a perturbation-based regularization - SmoothDARTS (SDARTS), to smooth the loss landscape and improve the generalizability of DARTS-based methods. In particular, our new formulations stabilize DARTS-based methods by either random smoothing or adversarial attack. The search trajectory on NAS-Bench-1Shot1 demonstrates the effectiveness of our approach and due to the improved stability, we achieve performance gain across various search spaces on 4 datasets. Furthermore, we mathematically show that SDARTS implicitly regularizes the Hessian norm of the validation loss, which accounts for a smoother loss landscape and improved performance.