Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
(Submitted on 12 Feb 2020 (v1), last revised 12 Jan 2021 (this version, v3))
Abstract: Differentiable architecture search (DARTS) is a prevailing NAS solution to identify architectures. Based on the continuous relaxation of the architecture space, DARTS learns a differentiable architecture weight and largely reduces the search cost. However, its stability has been challenged for yielding deteriorating architectures as the search proceeds. We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. Based on this observation, we propose a perturbation-based regularization - SmoothDARTS (SDARTS), to smooth the loss landscape and improve the generalizability of DARTS-based methods. In particular, our new formulations stabilize DARTS-based methods by either random smoothing or adversarial attack. The search trajectory on NAS-Bench-1Shot1 demonstrates the effectiveness of our approach and due to the improved stability, we achieve performance gain across various search spaces on 4 datasets. Furthermore, we mathematically show that SDARTS implicitly regularizes the Hessian norm of the validation loss, which accounts for a smoother loss landscape and improved performance.
Submission history
From: Xiangning Chen [view email][v1] Wed, 12 Feb 2020 23:46:58 GMT (4311kb,D)
[v2] Sat, 27 Jun 2020 21:56:42 GMT (4326kb,D)
[v3] Tue, 12 Jan 2021 19:17:24 GMT (4081kb,D)
Link back to: arXiv, form interface, contact.