Current browse context:
math.NA
Change to browse by:
References & Citations
Mathematics > Numerical Analysis
Title: Backpropagation through Back Substitution with a Backslash
(Submitted on 23 Feb 2023 (v1), last revised 31 Aug 2023 (this version, v2))
Abstract: We present a linear algebra formulation of backpropagation which allows the calculation of gradients by using a generically written ``backslash'' or Gaussian elimination on triangular systems of equations. Generally, the matrix elements are operators. This paper has three contributions: (i) it is of intellectual value to replace traditional treatments of automatic differentiation with a (left acting) operator theoretic, graph-based approach; (ii) operators can be readily placed in matrices in software in programming languages such as Julia as an implementation option; (iii) we introduce a novel notation, ``transpose dot'' operator ``$\{\}^{T_\bullet}$'' that allows for the reversal of operators.
We further demonstrate the elegance of the operators approach in a suitable programming language consisting of generic linear algebra operators such as Julia \cite{bezanson2017julia}, and that it is possible to realize this abstraction in code. Our implementation shows how generic linear algebra can allow operators as elements of matrices. In contrast to ``operator overloading,'' where backslash would normally have to be rewritten to take advantage of operators, with ``generic programming'' there is no such need.
Submission history
From: Yuyang Wang [view email][v1] Thu, 23 Feb 2023 23:51:44 GMT (22352kb,D)
[v2] Thu, 31 Aug 2023 01:11:10 GMT (5025kb,D)
Link back to: arXiv, form interface, contact.