Current browse context:
stat.ML
Change to browse by:
References & Citations
Statistics > Machine Learning
Title: Interpolation between Residual and Non-Residual Networks
(Submitted on 10 Jun 2020 (v1), last revised 16 Aug 2020 (this version, v4))
Abstract: Although ordinary differential equations (ODEs) provide insights for designing network architectures, its relationship with the non-residual convolutional neural networks (CNNs) is still unclear. In this paper, we present a novel ODE model by adding a damping term. It can be shown that the proposed model can recover both a ResNet and a CNN by adjusting an interpolation coefficient. Therefore, the damped ODE model provides a unified framework for the interpretation of residual and non-residual networks. The Lyapunov analysis reveals better stability of the proposed model, and thus yields robustness improvement of the learned networks. Experiments on a number of image classification benchmarks show that the proposed model substantially improves the accuracy of ResNet and ResNeXt over the perturbed inputs from both stochastic noise and adversarial attack methods. Moreover, the loss landscape analysis demonstrates the improved robustness of our method along the attack direction.
Submission history
From: Zonghan Yang [view email][v1] Wed, 10 Jun 2020 09:36:38 GMT (4285kb,D)
[v2] Fri, 26 Jun 2020 09:57:12 GMT (4308kb,D)
[v3] Mon, 3 Aug 2020 18:53:31 GMT (4308kb,D)
[v4] Sun, 16 Aug 2020 22:02:41 GMT (4305kb,D)
Link back to: arXiv, form interface, contact.