Technical Report ERGO 13-012

Higher-order reverse automatic differentiation with emphasis on the third-order
Robert Gower and Artur Gower

Abstract:

It is commonly assumed that calculating third order information is too expensive for most applications. But we show that the directional derivative of the Hessian ($D^3f(x)\cdot d$) can be calculated at a cost proportional to that of a state-of-the-art method for calculating the Hessian matrix. We do this by first presenting a simple procedure for designing high order reverse methods and applying it to deduce several methods including a reverse method that calculates $D^3f(x)\cdot d$. We have implemented this method taking into account symmetry and sparsity, and successfully calculated this derivative for functions with a million variables. These results indicate that the use of third order information in a general nonlinear solver, such as Halley-Chebyshev methods, could be a practical alternative to Newton's method.

Keywords:

Automatic differentiation, tensor calculation, nonlinear methods, reverse mode, Hessian matrix, Halley Chebyshev methods

Download:

ERGO-13-012.pdf

History:

Written: 22 June 2013

Status:

Submitted for publication