References & Citations
Computer Science > Programming Languages
Title: Efficient Dual-Numbers Reverse AD via Well-Known Program Transformations
(Submitted on 7 Jul 2022 (v1), last revised 21 Dec 2022 (this version, v2))
Abstract: Where dual-numbers forward-mode automatic differentiation (AD) pairs each scalar value with its tangent value, dual-numbers /reverse-mode/ AD attempts to achieve reverse AD using a similarly simple idea: by pairing each scalar value with a backpropagator function. Its correctness and efficiency on higher-order input languages have been analysed by Brunel, Mazza and Pagani, but this analysis used a custom operational semantics for which it is unclear whether it can be implemented efficiently. We take inspiration from their use of /linear factoring/ to optimise dual-numbers reverse-mode AD to an algorithm that has the correct complexity and enjoys an efficient implementation in a standard functional language with support for mutable arrays, such as Haskell. Aside from the linear factoring ingredient, our optimisation steps consist of well-known ideas from the functional programming community.
We demonstrate the practical use of our technique by providing a performant implementation that differentiates most of Haskell98.
Submission history
From: Tom Smeding [view email][v1] Thu, 7 Jul 2022 16:43:14 GMT (116kb,D)
[v2] Wed, 21 Dec 2022 20:08:40 GMT (373kb,D)
Link back to: arXiv, form interface, contact.