Baydin, Atilim Gunes and Pearlmutter, Barak A. and Siskind, Jeffrey Mark (2016) Tricks from Deep Learning. In: AD 2016 Conference: 7th International Conference on Algorithmic Differentiation, September 12-15, 2016, Oxford, U.K..

Preview

Download (299kB) | Preview

Abstract

The deep learning community has devised a diverse set of methods to make gradient optimization, using large datasets, of large and highly complex models with deeply cascaded nonlinearities, practical. Taken as a whole, these methods constitute a breakthrough, allowing computational structures which are quite wide, very deep, and with an enormous number and variety of free parameters to be effectively optimized. The result now dominates much of practical machine learning, with applications in machine translation, computer vision, and speech recognition. Many of these methods, viewed through the lens of algorithmic differentiation (AD), can be seen as either addressing issues with the gradient itself, or finding ways of achieving increased efficiency using tricks that are AD-related, but not provided by current AD systems. The goal of this paper is to explain not just those methods of most relevance to AD, but also the technical constraints and mindset which led to their discovery. After explaining this context, we present a "laundry list" of methods developed by the deep learning community. Two of these are discussed in further mathematical detail: a way to dramatically reduce the size of the tape when performing reverse-mode AD on a (theoretically) time-reversible process like an ODE integrator; and a new mathematical insight that allows for the implementation of a stochastic Newton's method.

Item Type:	Conference or Workshop Item (Paper)
Additional Information:	Extended abstract presented at the AD 2016 Conference, Sep 2016, Oxford UK. Cite as arXiv:1611.03777 [cs.LG]
Keywords:	machine learning; deep learning; algorithmic differentiation;
Academic Unit:	Faculty of Science and Engineering > Computer Science
Item ID:	8115
Depositing User:	Barak Pearlmutter
Date Deposited:	03 Apr 2017 15:37
Refereed:	Yes
Funders:	Science Foundation Ireland (SFI)
URI:	https://arxiv.org/abs/1611.03777
Use Licence:	This item is available under a Creative Commons Attribution Non Commercial Share Alike Licence (CC BY-NC-SA). Details of this licence are available here

Repository Staff Only(login required)

Item control page

Downloads

Origin of downloads