Dynamic Evaluation of Neural Sequence Models

Benjamin Krause; Emmanuel Kahembwe; Iain Murray; Steve Renals

Dynamic Evaluation of Neural Sequence Models

Benjamin Krause, Emmanuel Kahembwe, Iain Murray, Steve Renals

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We explore dynamic evaluation, where sequence models are adapted to the recent sequence history using gradient descent, assigning higher probabilities to re-occurring sequential patterns. We develop a dynamic evaluation approach that outperforms existing adaptation approaches in our comparisons. We apply dynamic evaluation to outperform all previous word-level perplexities on the Penn Treebank and WikiText-2 datasets (achieving 51.1 and 44.3 respectively) and all previous character-level cross-entropies on the text8 and Hutter Prize datasets (achieving 1.19 bits/char and 1.08 bits/char respectively).

Original language	English
Title of host publication	Proceedings of the 35th International Conference on Machine Learning
Editors	Jennifer Dy, Andreas Krause
Place of Publication	Stockholmsmässan, Stockholm Sweden
Publisher	PMLR
Pages	2766-2775
Number of pages	10
Publication status	Published - 15 Jul 2018
Event	Thirty-fifth International Conference on Machine Learning - Stockholmsmässan, Stockholm, Sweden Duration: 10 Jul 2018 → 15 Jul 2018 https://icml.cc/ https://icml.cc/

Publication series

Name	Proceedings of Machine Learning Research
Publisher	PMLR
Volume	80
ISSN (Electronic)	2640-3498

Conference

Conference	Thirty-fifth International Conference on Machine Learning
Abbreviated title	ICML 2018
Country/Territory	Sweden
City	Stockholm
Period	10/07/18 → 15/07/18
Internet address	https://icml.cc/ https://icml.cc/

Access to Document

Dynamic_evaluation_of_neural_sequence_models_2018Accepted author manuscript, 327 KB
Dynamic Evaluation_KRAUSE_DOA11052018_VOR_CC-BYFinal published version, 327 KBLicence: Creative Commons: Attribution (CC-BY)

http://proceedings.mlr.press/v80/krause18a.html

SUMMA - Scalable Understanding of Mulitingual Media
Renals, S., Birch-Mayne, A. & Cohen, S.
EU government bodies
1/02/16 → 31/01/19
Project: Research

Stephen Renals
- School of Informatics - UoE Honorary staff
Person: Affiliated Independent Researcher

Cite this

@inproceedings{53648a167113418093a79aa46b960cba,

title = "Dynamic Evaluation of Neural Sequence Models",

abstract = "We explore dynamic evaluation, where sequence models are adapted to the recent sequence history using gradient descent, assigning higher probabilities to re-occurring sequential patterns. We develop a dynamic evaluation approach that outperforms existing adaptation approaches in our comparisons. We apply dynamic evaluation to outperform all previous word-level perplexities on the Penn Treebank and WikiText-2 datasets (achieving 51.1 and 44.3 respectively) and all previous character-level cross-entropies on the text8 and Hutter Prize datasets (achieving 1.19 bits/char and 1.08 bits/char respectively).",

author = "Benjamin Krause and Emmanuel Kahembwe and Iain Murray and Steve Renals",

year = "2018",

month = jul,

day = "15",

language = "English",

series = "Proceedings of Machine Learning Research",

publisher = "PMLR",

pages = "2766--2775",

editor = "Jennifer Dy and Andreas Krause",

booktitle = "Proceedings of the 35th International Conference on Machine Learning",

note = "Thirty-fifth International Conference on Machine Learning, ICML 2018 ; Conference date: 10-07-2018 Through 15-07-2018",

url = "https://icml.cc/, https://icml.cc/",

}

Krause, B, Kahembwe, E, Murray, I & Renals, S 2018, Dynamic Evaluation of Neural Sequence Models. in J Dy & A Krause (eds), Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80, PMLR, Stockholmsmässan, Stockholm Sweden, pp. 2766-2775, Thirty-fifth International Conference on Machine Learning, Stockholm, Sweden, 10/07/18. <http://proceedings.mlr.press/v80/krause18a.html>

Dynamic Evaluation of Neural Sequence Models. / Krause, Benjamin; Kahembwe, Emmanuel; Murray, Iain et al.
Proceedings of the 35th International Conference on Machine Learning. ed. / Jennifer Dy; Andreas Krause. Stockholmsmässan, Stockholm Sweden: PMLR, 2018. p. 2766-2775 (Proceedings of Machine Learning Research; Vol. 80).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Dynamic Evaluation of Neural Sequence Models

AU - Krause, Benjamin

AU - Kahembwe, Emmanuel

AU - Murray, Iain

AU - Renals, Steve

PY - 2018/7/15

Y1 - 2018/7/15

N2 - We explore dynamic evaluation, where sequence models are adapted to the recent sequence history using gradient descent, assigning higher probabilities to re-occurring sequential patterns. We develop a dynamic evaluation approach that outperforms existing adaptation approaches in our comparisons. We apply dynamic evaluation to outperform all previous word-level perplexities on the Penn Treebank and WikiText-2 datasets (achieving 51.1 and 44.3 respectively) and all previous character-level cross-entropies on the text8 and Hutter Prize datasets (achieving 1.19 bits/char and 1.08 bits/char respectively).

AB - We explore dynamic evaluation, where sequence models are adapted to the recent sequence history using gradient descent, assigning higher probabilities to re-occurring sequential patterns. We develop a dynamic evaluation approach that outperforms existing adaptation approaches in our comparisons. We apply dynamic evaluation to outperform all previous word-level perplexities on the Penn Treebank and WikiText-2 datasets (achieving 51.1 and 44.3 respectively) and all previous character-level cross-entropies on the text8 and Hutter Prize datasets (achieving 1.19 bits/char and 1.08 bits/char respectively).

M3 - Conference contribution

T3 - Proceedings of Machine Learning Research

SP - 2766

EP - 2775

BT - Proceedings of the 35th International Conference on Machine Learning

A2 - Dy, Jennifer

A2 - Krause, Andreas

PB - PMLR

CY - Stockholmsmässan, Stockholm Sweden

T2 - Thirty-fifth International Conference on Machine Learning

Y2 - 10 July 2018 through 15 July 2018

ER -

Dynamic Evaluation of Neural Sequence Models

Abstract

Publication series

Conference

Access to Document

Fingerprint

Projects

SUMMA - Scalable Understanding of Mulitingual Media

Profiles

Stephen Renals

Cite this