We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Mathematics > Numerical Analysis

Title: BACKpropagation through BACK substitution with a BACKslash

Abstract: We present a linear algebra formulation of backpropagation which allows the calculation of gradients by using a generically written ``backslash'' or Gaussian elimination on triangular systems of equations. Generally the matrix elements are operators. This paper has three contributions:
1. It is of intellectual value to replace traditional treatments of automatic differentiation with a (left acting) operator theoretic, graph-based approach.
2. Operators can be readily placed in matrices in software in programming languages such as Ju lia as an implementation option.
3. We introduce a novel notation, ``transpose dot'' operator ``$\{\}^{T_\bullet}$'' that allows the reversal of operators.
We demonstrate the elegance of the operators approach in a suitable programming language consisting of generic linear algebra operators such as Julia \cite{bezanson2017julia}, and that it is possible to realize this abstraction in code. Our implementation shows how generic linear algebra can allow operators as elements of matrices, and without rewriting any code, the software carries through to completion giving the correct answer.
Comments: 21 pages
Subjects: Numerical Analysis (math.NA); Machine Learning (cs.LG)
Cite as: arXiv:2303.15449 [math.NA]
  (or arXiv:2303.15449v1 [math.NA] for this version)

Submission history

From: Yuyang Wang [view email]
[v1] Thu, 23 Feb 2023 23:51:44 GMT (22352kb,D)

Link back to: arXiv, form interface, contact.