Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training

Alham Fikri Aji; Kenneth Heafield; Nikolay Bogoychev

doi:10.18653/v1/D19-1373

Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training

Alham Fikri Aji, Kenneth Heafield, Nikolay Bogoychev

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

One way to reduce network traffic in multinode data-parallel stochastic gradient descent is to only exchange the largest gradients. However, doing so damages the gradient and degrades the model’s performance. Tranformer models degrade dramatically while the impact on RNNs is smaller. We restore gradient quality by combining the compressed global gradient with the node’s locally computed uncompressed gradient. Neural machine translation experiments show that Transformer convergence is restored while RNNs converge faster. With our method, training on 4 nodes converges up to 1.5x as fast as with uncompressed gradients and scales 3.5x relative to single-node training

Original language	English
Title of host publication	Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing
Place of Publication	Hong Kong
Publisher	Association for Computational Linguistics
Pages	3624–3629
Number of pages	6
ISBN (Print)	978-1-950737-90-1
DOIs	https://doi.org/10.18653/v1/D19-1373
Publication status	Published - 4 Nov 2019
Event	2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing - Hong Kong, Hong Kong Duration: 3 Nov 2019 → 7 Nov 2019 https://www.emnlp-ijcnlp2019.org/

Conference

Conference	2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing
Abbreviated title	EMNLP-IJCNLP 2019
Country/Territory	Hong Kong
City	Hong Kong
Period	3/11/19 → 7/11/19
Internet address	https://www.emnlp-ijcnlp2019.org/

Access to Document

10.18653/v1/D19-1373Licence: Creative Commons: Attribution (CC-BY)

glocalAccepted author manuscript, 271 KBLicence: Creative Commons: Attribution (CC-BY)
Combining Global Sparse_AJI_DOA04112019_VOR_CC-BYFinal published version, 350 KBLicence: Creative Commons: Attribution (CC-BY)

Cite this

Aji, A. F., Heafield, K., & Bogoychev, N. (2019). Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (pp. 3624–3629). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1373

Aji, Alham Fikri ; Heafield, Kenneth ; Bogoychev, Nikolay. / Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing. Hong Kong : Association for Computational Linguistics, 2019. pp. 3624–3629

@inproceedings{ca73d63debcb40f8938722237c63b2a7,

title = "Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training",

abstract = "One way to reduce network traffic in multinode data-parallel stochastic gradient descent is to only exchange the largest gradients. However, doing so damages the gradient and degrades the model{\textquoteright}s performance. Tranformer models degrade dramatically while the impact on RNNs is smaller. We restore gradient quality by combining the compressed global gradient with the node{\textquoteright}s locally computed uncompressed gradient. Neural machine translation experiments show that Transformer convergence is restored while RNNs converge faster. With our method, training on 4 nodes converges up to 1.5x as fast as with uncompressed gradients and scales 3.5x relative to single-node training",

author = "Aji, \{Alham Fikri\} and Kenneth Heafield and Nikolay Bogoychev",

year = "2019",

month = nov,

day = "4",

doi = "10.18653/v1/D19-1373",

language = "English",

isbn = "978-1-950737-90-1",

pages = "3624–3629",

booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing",

publisher = "Association for Computational Linguistics",

note = "2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019 ; Conference date: 03-11-2019 Through 07-11-2019",

url = "https://www.emnlp-ijcnlp2019.org/",

}

Aji, AF, Heafield, K & Bogoychev, N 2019, Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training. in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, Hong Kong, pp. 3624–3629, 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Hong Kong, Hong Kong, 3/11/19. https://doi.org/10.18653/v1/D19-1373

Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training. / Aji, Alham Fikri; Heafield, Kenneth; Bogoychev, Nikolay.
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing. Hong Kong: Association for Computational Linguistics, 2019. p. 3624–3629.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training

AU - Aji, Alham Fikri

AU - Heafield, Kenneth

AU - Bogoychev, Nikolay

PY - 2019/11/4

Y1 - 2019/11/4

N2 - One way to reduce network traffic in multinode data-parallel stochastic gradient descent is to only exchange the largest gradients. However, doing so damages the gradient and degrades the model’s performance. Tranformer models degrade dramatically while the impact on RNNs is smaller. We restore gradient quality by combining the compressed global gradient with the node’s locally computed uncompressed gradient. Neural machine translation experiments show that Transformer convergence is restored while RNNs converge faster. With our method, training on 4 nodes converges up to 1.5x as fast as with uncompressed gradients and scales 3.5x relative to single-node training

AB - One way to reduce network traffic in multinode data-parallel stochastic gradient descent is to only exchange the largest gradients. However, doing so damages the gradient and degrades the model’s performance. Tranformer models degrade dramatically while the impact on RNNs is smaller. We restore gradient quality by combining the compressed global gradient with the node’s locally computed uncompressed gradient. Neural machine translation experiments show that Transformer convergence is restored while RNNs converge faster. With our method, training on 4 nodes converges up to 1.5x as fast as with uncompressed gradients and scales 3.5x relative to single-node training

U2 - 10.18653/v1/D19-1373

DO - 10.18653/v1/D19-1373

M3 - Conference contribution

SN - 978-1-950737-90-1

SP - 3624

EP - 3629

BT - Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing

PB - Association for Computational Linguistics

CY - Hong Kong

T2 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing

Y2 - 3 November 2019 through 7 November 2019

ER -

Aji AF, Heafield K, Bogoychev N. Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing. Hong Kong: Association for Computational Linguistics. 2019. p. 3624–3629 doi: 10.18653/v1/D19-1373

Combining Global Sparse Gradients with Local Gradients in Distributed Neural Network Training

Abstract

Conference

Access to Document

Fingerprint

Cite this