Projects per year
Abstract / Description of output
Stochastic Gradient Descent (SGD) based training of neural networks with
a large learning rate or a small batch-size typically ends in well-generalizing,
flat regions of the weight space, as indicated by small eigenvalues of the
Hessian of the training loss. However, the curvature along the SGD trajectory
is poorly understood. An empirical investigation shows that initially SGD
visits increasingly sharp regions, reaching a maximum sharpness determined
by both the learning rate and the batch-size of SGD. When studying the
SGD dynamics in relation to the sharpest directions in this initial phase, we
find that the SGD step is large compared to the curvature and commonly
fails to minimize the loss along the sharpest directions. Furthermore, using
a reduced learning rate along these directions can improve training speed
while leading to both sharper and better generalizing solutions compared
to vanilla SGD. In summary, our analysis of the dynamics of SGD in the
subspace of the sharpest directions shows that they influence the regions
that SGD steers to (where larger learning rate or smaller batch size result
in wider regions visited), the overall training speed, and the generalization
ability of the final model.
Original language | English |
---|---|
Number of pages | 19 |
Publication status | Published - 2019 |
Event | Seventh International Conference on Learning Representations - New Orleans, United States Duration: 6 May 2019 → 9 May 2019 https://iclr.cc/ |
Conference
Conference | Seventh International Conference on Learning Representations |
---|---|
Abbreviated title | ICLR 2019 |
Country/Territory | United States |
City | New Orleans |
Period | 6/05/19 → 9/05/19 |
Internet address |
Fingerprint
Dive into the research topics of 'On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length'. Together they form a unique fingerprint.Projects
- 1 Finished
-
Bonseyes - Platform for Open Development of Systems of Artificial Intelligence
1/12/16 → 31/01/20
Project: Research