On the Efficiency of Recurrent Neural Network Optimization Algorithms

Ben  Krause; Liang Lu; Iain Murray; Steve Renals

On the Efficiency of Recurrent Neural Network Optimization Algorithms

Ben Krause, Liang Lu, Iain Murray, Steve Renals

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This study compares the sequential and parallel efficiency of training Recurrent Neural Networks (RNNs) with Hessian-free optimization versus a gradient descent variant. Experiments are performed using the long short term memory (LSTM)
architecture and the newly proposed multiplicative LSTM (mLSTM) architecture.
Results demonstrate a number of insights into these architectures and optimization
algorithms, including that Hessian-free optimization has the potential for large
efficiency gains in a highly parallel setup.

Original language	English
Title of host publication	OPT2015 Optimization for Machine Learning at the Neural Information Processing Systems Conference, 2015
Number of pages	5
Publication status	Published - Dec 2015

Access to Document

OPT2015_paper_25Accepted author manuscript, 201 KB

http://opt-ml.org/papers/OPT2015_paper_25.pdf

Cite this

@inproceedings{1d513f350f22466f8447b71a3a930591,

title = "On the Efficiency of Recurrent Neural Network Optimization Algorithms",

abstract = "This study compares the sequential and parallel efficiency of training Recurrent Neural Networks (RNNs) with Hessian-free optimization versus a gradient descent variant. Experiments are performed using the long short term memory (LSTM) architecture and the newly proposed multiplicative LSTM (mLSTM) architecture. Results demonstrate a number of insights into these architectures and optimization algorithms, including that Hessian-free optimization has the potential for large efficiency gains in a highly parallel setup.",

author = "Ben Krause and Liang Lu and Iain Murray and Steve Renals",

year = "2015",

month = dec,

language = "English",

booktitle = "OPT2015 Optimization for Machine Learning at the Neural Information Processing Systems Conference, 2015",

}

TY - GEN

T1 - On the Efficiency of Recurrent Neural Network Optimization Algorithms

AU - Krause, Ben

AU - Lu, Liang

AU - Murray, Iain

AU - Renals, Steve

PY - 2015/12

Y1 - 2015/12

N2 - This study compares the sequential and parallel efficiency of training Recurrent Neural Networks (RNNs) with Hessian-free optimization versus a gradient descent variant. Experiments are performed using the long short term memory (LSTM) architecture and the newly proposed multiplicative LSTM (mLSTM) architecture. Results demonstrate a number of insights into these architectures and optimization algorithms, including that Hessian-free optimization has the potential for large efficiency gains in a highly parallel setup.

AB - This study compares the sequential and parallel efficiency of training Recurrent Neural Networks (RNNs) with Hessian-free optimization versus a gradient descent variant. Experiments are performed using the long short term memory (LSTM) architecture and the newly proposed multiplicative LSTM (mLSTM) architecture. Results demonstrate a number of insights into these architectures and optimization algorithms, including that Hessian-free optimization has the potential for large efficiency gains in a highly parallel setup.

M3 - Conference contribution

BT - OPT2015 Optimization for Machine Learning at the Neural Information Processing Systems Conference, 2015

ER -

On the Efficiency of Recurrent Neural Network Optimization Algorithms

Abstract

Access to Document

Fingerprint

Cite this