Pearlmutter, Barak A. and Siskind, Jeffrey Mark (2008) Reverse-Mode AD in a Functional Framework: Lambda the Ultimate Backpropagator. ACM Transactions on Programming Languages and Systems, 30 (2). 7.1-7.36. ISSN 0164-0925

Download (354kB)

Abstract

We show that reverse-mode AD (Automatic Differentiation)—a generalized gradient-calculation operator—can be incorporated as a first-class function in an augmented lambda calculus, and therefore into a functional-programming language. Closure is achieved, in that the new operator can be applied to any expression in the augmented language, yielding an expression in that language. This requires the resolution of two major technical issues: (a) how to transform nested lambda expressions, including those with free-variable references, and (b) how to support self application of the AD machinery. AD transformations preserve certain complexity properties, among them that the reverse phase of the reverse-mode AD transformation of a function have the same temporal complexity as the original untransformed function. First-class unrestricted AD operators increase the expressive power available to the numeric programmer, and may have significant practical implications for the construction of numeric software that is robust, modular, concise, correct, and efficient.

Item Type:	Article
Additional Information:	The work of B. A. Pearlmutter was supported, in part, by Science Foundation Ireland grant 00/PI.1/C067 and the Higher Education Authority of Ireland. The work of J. M. Siskind was supported, in part, by National Science Foundation (NSF) grant CCF-0438806. © ACM, 2008. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Programming Languages and Systems (TOPLAS) Volume 30 Issue 2, March 2008, http://doi.acm.org/10.1145/1330017.1330018
Keywords:	closures; derivatives; forward-mode AD; higher-order AD; higher-order functional languages; Jacobian; program transformation; reflection;
Academic Unit:	Faculty of Science and Engineering > Computer Science Faculty of Science and Engineering > Research Institutes > Hamilton Institute
Item ID:	3504
Depositing User:	Barak Pearlmutter
Date Deposited:	29 Feb 2012 14:25
Journal or Publication Title:	ACM Transactions on Programming Languages and Systems
Publisher:	Association for Computing Machinery (ACM)
Refereed:	Yes
Funders:	Higher Education Authority of Ireland, National Science Foundation (NSF), Science Foundation Ireland
URI:	http://toplas.acm.org/
Use Licence:	This item is available under a Creative Commons Attribution Non Commercial Share Alike Licence (CC BY-NC-SA). Details of this licence are available here

Repository Staff Only(login required)

Item control page

Downloads

Origin of downloads