Current browse context:
math
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Improved architectures and training algorithms for deep operator networks
(Submitted on 4 Oct 2021 (v1), last revised 11 Oct 2021 (this version, v2))
Abstract: Operator learning techniques have recently emerged as a powerful tool for learning maps between infinite-dimensional Banach spaces. Trained under appropriate constraints, they can also be effective in learning the solution operator of partial differential equations (PDEs) in an entirely self-supervised manner. In this work we analyze the training dynamics of deep operator networks (DeepONets) through the lens of Neural Tangent Kernel (NTK) theory, and reveal a bias that favors the approximation of functions with larger magnitudes. To correct this bias we propose to adaptively re-weight the importance of each training example, and demonstrate how this procedure can effectively balance the magnitude of back-propagated gradients during training via gradient descent. We also propose a novel network architecture that is more resilient to vanishing gradient pathologies. Taken together, our developments provide new insights into the training of DeepONets and consistently improve their predictive accuracy by a factor of 10-50x, demonstrated in the challenging setting of learning PDE solution operators in the absence of paired input-output observations. All code and data accompanying this manuscript are publicly available at \url{this https URL}
Submission history
From: Sifan Wang [view email][v1] Mon, 4 Oct 2021 18:34:41 GMT (6043kb,D)
[v2] Mon, 11 Oct 2021 21:40:57 GMT (6042kb,D)
Link back to: arXiv, form interface, contact.