Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio

Stanislaw Jastrzębski; Zachary  Kenton; Devansh Arpit; Nicolas  Ballas; Asja Fischer; Yoshua Bengio; Amos Storkey

doi:10.1007/978-3-030-01424-7_39

Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio

Stanislaw Jastrzębski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, Amos Storkey

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We show that the dynamics and convergence properties of SGD are set by the ratio of learning rate to batch size. We observe that this ratio is a key determinant of the generalization error, which we suggest is mediated by controlling the width of the final minima found by SGD. We verify our analysis experimentally on a range of deep neural networks and datasets.

Original language	English
Title of host publication	Proceedings of 27th International Conference on Artificial Neural Networks
Place of Publication	Rhodes, Greece
Publisher	Springer
Pages	392-402
Number of pages	10
ISBN (Electronic)	978-3-030-01424-7
ISBN (Print)	978-3-030-01423-0
DOIs	https://doi.org/10.1007/978-3-030-01424-7_39
Publication status	Published - Oct 2018
Event	27th International Conference on Artificial Neural Networks - Rhodes, Greece Duration: 4 Oct 2018 → 7 Oct 2018 https://e-nns.org/icann2018/

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer, Cham
Volume	11141
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Name	Theoretical Computer Science and General Issues
Volume	11141

Conference

Conference	27th International Conference on Artificial Neural Networks
Abbreviated title	ICANN 2018
Country/Territory	Greece
City	Rhodes
Period	4/10/18 → 7/10/18
Internet address	https://e-nns.org/icann2018/

Access to Document

10.1007/978-3-030-01424-7_39

width of minima reached by stochastic gradient descent is influenced by learning rate to batch size ratioAccepted author manuscript, 537 KB

https://link.springer.com/chapter/10.1007/978-3-030-01424-7_39

Bonseyes - Platform for Open Development of Systems of Artificial Intelligence
Storkey, A. (Principal Investigator) & O'Boyle, M. (Co-investigator)
EU government bodies
1/12/16 → 31/01/20
Project: Research

Cite this

Jastrzębski, S., Kenton, Z., Arpit, D., Ballas, N., Fischer, A., Bengio, Y., & Storkey, A. (2018). Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio. In Proceedings of 27th International Conference on Artificial Neural Networks (pp. 392-402). (Lecture Notes in Computer Science; Vol. 11141), (Theoretical Computer Science and General Issues; Vol. 11141). Springer. https://doi.org/10.1007/978-3-030-01424-7_39

Jastrzębski, Stanislaw ; Kenton, Zachary ; Arpit, Devansh et al. / Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio. Proceedings of 27th International Conference on Artificial Neural Networks. Rhodes, Greece : Springer, 2018. pp. 392-402 (Lecture Notes in Computer Science). (Theoretical Computer Science and General Issues).

@inproceedings{1b1d210aefed44b78907c1506f70a64d,

title = "Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio",

abstract = "We show that the dynamics and convergence properties of SGD are set by the ratio of learning rate to batch size. We observe that this ratio is a key determinant of the generalization error, which we suggest is mediated by controlling the width of the final minima found by SGD. We verify our analysis experimentally on a range of deep neural networks and datasets.",

author = "Stanislaw Jastrz{\c e}bski and Zachary Kenton and Devansh Arpit and Nicolas Ballas and Asja Fischer and Yoshua Bengio and Amos Storkey",

year = "2018",

month = oct,

doi = "10.1007/978-3-030-01424-7\_39",

language = "English",

isbn = "978-3-030-01423-0",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "392--402",

booktitle = "Proceedings of 27th International Conference on Artificial Neural Networks",

address = "United Kingdom",

note = "27th International Conference on Artificial Neural Networks , ICANN 2018 ; Conference date: 04-10-2018 Through 07-10-2018",

url = "https://e-nns.org/icann2018/",

}

Jastrzębski, S, Kenton, Z, Arpit, D, Ballas, N, Fischer, A, Bengio, Y & Storkey, A 2018, Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio. in Proceedings of 27th International Conference on Artificial Neural Networks. Lecture Notes in Computer Science, vol. 11141, Theoretical Computer Science and General Issues, vol. 11141, Springer, Rhodes, Greece, pp. 392-402, 27th International Conference on Artificial Neural Networks , Rhodes, Greece, 4/10/18. https://doi.org/10.1007/978-3-030-01424-7_39

Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio. / Jastrzębski, Stanislaw; Kenton, Zachary ; Arpit, Devansh et al.
Proceedings of 27th International Conference on Artificial Neural Networks. Rhodes, Greece: Springer, 2018. p. 392-402 (Lecture Notes in Computer Science; Vol. 11141), (Theoretical Computer Science and General Issues; Vol. 11141).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio

AU - Jastrzębski, Stanislaw

AU - Kenton, Zachary

AU - Arpit, Devansh

AU - Ballas, Nicolas

AU - Fischer, Asja

AU - Bengio, Yoshua

AU - Storkey, Amos

PY - 2018/10

Y1 - 2018/10

N2 - We show that the dynamics and convergence properties of SGD are set by the ratio of learning rate to batch size. We observe that this ratio is a key determinant of the generalization error, which we suggest is mediated by controlling the width of the final minima found by SGD. We verify our analysis experimentally on a range of deep neural networks and datasets.

AB - We show that the dynamics and convergence properties of SGD are set by the ratio of learning rate to batch size. We observe that this ratio is a key determinant of the generalization error, which we suggest is mediated by controlling the width of the final minima found by SGD. We verify our analysis experimentally on a range of deep neural networks and datasets.

U2 - 10.1007/978-3-030-01424-7_39

DO - 10.1007/978-3-030-01424-7_39

M3 - Conference contribution

SN - 978-3-030-01423-0

T3 - Lecture Notes in Computer Science

SP - 392

EP - 402

BT - Proceedings of 27th International Conference on Artificial Neural Networks

PB - Springer

CY - Rhodes, Greece

T2 - 27th International Conference on Artificial Neural Networks

Y2 - 4 October 2018 through 7 October 2018

ER -

Jastrzębski S, Kenton Z, Arpit D, Ballas N, Fischer A, Bengio Y et al. Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio. In Proceedings of 27th International Conference on Artificial Neural Networks. Rhodes, Greece: Springer. 2018. p. 392-402. (Lecture Notes in Computer Science). (Theoretical Computer Science and General Issues). doi: 10.1007/978-3-030-01424-7_39

Width of Minima Reached by Stochastic Gradient Descent is Influenced by Learning Rate to Batch Size Ratio

Abstract

Publication series

Conference

Access to Document

Fingerprint

Projects

Bonseyes - Platform for Open Development of Systems of Artificial Intelligence

Cite this