MURAL - Maynooth University Research Archive Library

    Tricks from Deep Learning

    Baydin, Atilim Gunes and Pearlmutter, Barak A. and Siskind, Jeffrey Mark (2016) Tricks from Deep Learning. In: AD 2016 Conference: 7th International Conference on Algorithmic Differentiation, September 12-15, 2016, Oxford, U.K..

    Download (299kB) | Preview

    Share your research

    Twitter Facebook LinkedIn GooglePlus Email more...

    Add this article to your Mendeley library


    The deep learning community has devised a diverse set of methods to make gradient optimization, using large datasets, of large and highly complex models with deeply cascaded nonlinearities, practical. Taken as a whole, these methods constitute a breakthrough, allowing computational structures which are quite wide, very deep, and with an enormous number and variety of free parameters to be effectively optimized. The result now dominates much of practical machine learning, with applications in machine translation, computer vision, and speech recognition. Many of these methods, viewed through the lens of algorithmic differentiation (AD), can be seen as either addressing issues with the gradient itself, or finding ways of achieving increased efficiency using tricks that are AD-related, but not provided by current AD systems. The goal of this paper is to explain not just those methods of most relevance to AD, but also the technical constraints and mindset which led to their discovery. After explaining this context, we present a "laundry list" of methods developed by the deep learning community. Two of these are discussed in further mathematical detail: a way to dramatically reduce the size of the tape when performing reverse-mode AD on a (theoretically) time-reversible process like an ODE integrator; and a new mathematical insight that allows for the implementation of a stochastic Newton's method.

    Item Type: Conference or Workshop Item (Paper)
    Additional Information: Extended abstract presented at the AD 2016 Conference, Sep 2016, Oxford UK. Cite as arXiv:1611.03777 [cs.LG]
    Keywords: machine learning; deep learning; algorithmic differentiation;
    Academic Unit: Faculty of Science and Engineering > Computer Science
    Item ID: 8115
    Depositing User: Barak Pearlmutter
    Date Deposited: 03 Apr 2017 15:37
    Refereed: Yes
    Funders: Science Foundation Ireland (SFI)
    Use Licence: This item is available under a Creative Commons Attribution Non Commercial Share Alike Licence (CC BY-NC-SA). Details of this licence are available here

    Repository Staff Only(login required)

    View Item Item control page


    Downloads per month over past year

    Origin of downloads