Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Noisy Differentiable Architecture Search
(Submitted on 7 May 2020 (v1), last revised 17 Oct 2021 (this version, v3))
Abstract: Simplicity is the ultimate sophistication. Differentiable Architecture Search (DARTS) has now become one of the mainstream paradigms of neural architecture search. However, it largely suffers from the well-known performance collapse issue due to the aggregation of skip connections. It is thought to have overly benefited from the residual structure which accelerates the information flow. To weaken this impact, we propose to inject unbiased random noise to impede the flow. We name this novel approach NoisyDARTS. In effect, a network optimizer should perceive this difficulty at each training step and refrain from overshooting, especially on skip connections. In the long run, since we add no bias to the gradient in terms of expectation, it is still likely to converge to the right solution area. We also prove that the injected noise plays a role in smoothing the loss landscape, which makes the optimization easier. Our method features extreme simplicity and acts as a new strong baseline. We perform extensive experiments across various search spaces, datasets, and tasks, where we robustly achieve state-of-the-art results. Our code is available at this https URL
Submission history
From: Bo Zhang [view email][v1] Thu, 7 May 2020 15:53:52 GMT (435kb,D)
[v2] Tue, 19 May 2020 14:42:33 GMT (416kb,D)
[v3] Sun, 17 Oct 2021 14:57:46 GMT (6448kb,D)
Link back to: arXiv, form interface, contact.