Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Noisy Differentiable Architecture Search
(Submitted on 7 May 2020 (v1), revised 19 May 2020 (this version, v2), latest version 17 Oct 2021 (v3))
Abstract: Simplicity is the ultimate sophistication. Differentiable Architecture Search (DARTS) has now become one of the mainstream paradigms of neural architecture search. However, it largely suffers from several disturbing factors of optimization process whose results are unstable to reproduce. FairDARTS points out that skip connections natively have an unfair advantage in exclusive competition which primarily leads to dramatic performance collapse. While FairDARTS turns the unfair competition into a collaborative one, we instead impede such unfair advantage by injecting unbiased random noise into skip operations' output. In effect, the optimizer should perceive this difficulty at each training step and refrain from overshooting on skip connections, but in a long run it still converges to the right solution area since no bias is added to the gradient. We name this novel approach as NoisyDARTS. Our experiments on CIFAR-10 and ImageNet attest that it can effectively break the skip connection's unfair advantage and yield better performance. It generates a series of models that achieve state-of-the-art results on both datasets. Code will be made available at this https URL
Submission history
From: Bo Zhang [view email][v1] Thu, 7 May 2020 15:53:52 GMT (435kb,D)
[v2] Tue, 19 May 2020 14:42:33 GMT (416kb,D)
[v3] Sun, 17 Oct 2021 14:57:46 GMT (6448kb,D)
Link back to: arXiv, form interface, contact.