Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation

Brian Thompson; Jeremy Gwinnup; Huda Khayrallah; Kevin Duh; Philipp Koehn

doi:10.18653/v1/n19-1209

Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation

Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, Philipp Koehn

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Continued training is an effective method for domain adaptation in neural machine translation. However, in-domain gains from adaptation come at the expense of general-domain performance. In this work, we interpret the drop in general-domain performance as catastrophic forgetting of general-domain knowledge. To mitigate it, we adapt Elastic Weight Consolidation (EWC)—a machine learning method for learning a new task without forgetting previous tasks. Our method retains the majority of general-domain performance lost in continued training without degrading indomain performance, outperforming the previous state-of-the-art. We also explore the full range of general-domain performance available when some in-domain degradation is acceptable.

Original language	English
Title of host publication	Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)
Editors	Jill Burstein, Christy Doran, Thamar Solorio
Place of Publication	Minneapolis, Minnesota
Publisher	Association for Computational Linguistics
Pages	2062–2068
Number of pages	7
Volume	1
DOIs	https://doi.org/10.18653/v1/n19-1209
Publication status	Published - 7 Jun 2019
Event	2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics - Minneapolis, United States Duration: 2 Jun 2019 → 7 Jun 2019 https://naacl2019.org/

Conference

Conference	2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Abbreviated title	NAACL-HLT 2019
Country/Territory	United States
City	Minneapolis
Period	2/06/19 → 7/06/19
Internet address	https://naacl2019.org/

Access to Document

10.18653/v1/n19-1209Licence: Creative Commons: Attribution (CC-BY)

Overcoming Catastrophic Forgetting_THOMPSON_DoA220219_AFVAccepted author manuscript, 578 KB

https://www.aclweb.org/anthology/papers/N/N19/N19-1209/Licence: Creative Commons: Attribution (CC-BY)

Cite this

Thompson, B., Gwinnup, J., Khayrallah, H., Duh, K., & Koehn, P. (2019). Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) (Vol. 1, pp. 2062–2068). Association for Computational Linguistics. https://doi.org/10.18653/v1/n19-1209

Thompson, Brian ; Gwinnup, Jeremy ; Khayrallah, Huda ; Duh, Kevin ; Koehn, Philipp. / Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). editor / Jill Burstein ; Christy Doran ; Thamar Solorio. Vol. 1 Minneapolis, Minnesota : Association for Computational Linguistics, 2019. pp. 2062–2068

@inproceedings{7d3bbbee47044504b839c2718d312258,

title = "Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation",

abstract = "Continued training is an effective method for domain adaptation in neural machine translation. However, in-domain gains from adaptation come at the expense of general-domain performance. In this work, we interpret the drop in general-domain performance as catastrophic forgetting of general-domain knowledge. To mitigate it, we adapt Elastic Weight Consolidation (EWC)—a machine learning method for learning a new task without forgetting previous tasks. Our method retains the majority of general-domain performance lost in continued training without degrading indomain performance, outperforming the previous state-of-the-art. We also explore the full range of general-domain performance available when some in-domain degradation is acceptable.",

author = "Brian Thompson and Jeremy Gwinnup and Huda Khayrallah and Kevin Duh and Philipp Koehn",

year = "2019",

month = jun,

day = "7",

doi = "10.18653/v1/n19-1209",

language = "English",

volume = "1",

pages = "2062–2068",

editor = "Jill Burstein and Christy Doran and Solorio, {Thamar }",

booktitle = "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)",

publisher = "Association for Computational Linguistics",

note = "2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT 2019 ; Conference date: 02-06-2019 Through 07-06-2019",

url = "https://naacl2019.org/",

}

Thompson, B, Gwinnup, J, Khayrallah, H, Duh, K & Koehn, P 2019, Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation. in J Burstein, C Doran & T Solorio (eds), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). vol. 1, Association for Computational Linguistics, Minneapolis, Minnesota, pp. 2062–2068, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, Minnesota, United States, 2/06/19. https://doi.org/10.18653/v1/n19-1209

Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation. / Thompson, Brian; Gwinnup, Jeremy; Khayrallah, Huda; Duh, Kevin; Koehn, Philipp.

Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). ed. / Jill Burstein; Christy Doran; Thamar Solorio. Vol. 1 Minneapolis, Minnesota : Association for Computational Linguistics, 2019. p. 2062–2068.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation

AU - Thompson, Brian

AU - Gwinnup, Jeremy

AU - Khayrallah, Huda

AU - Duh, Kevin

AU - Koehn, Philipp

PY - 2019/6/7

Y1 - 2019/6/7

N2 - Continued training is an effective method for domain adaptation in neural machine translation. However, in-domain gains from adaptation come at the expense of general-domain performance. In this work, we interpret the drop in general-domain performance as catastrophic forgetting of general-domain knowledge. To mitigate it, we adapt Elastic Weight Consolidation (EWC)—a machine learning method for learning a new task without forgetting previous tasks. Our method retains the majority of general-domain performance lost in continued training without degrading indomain performance, outperforming the previous state-of-the-art. We also explore the full range of general-domain performance available when some in-domain degradation is acceptable.

AB - Continued training is an effective method for domain adaptation in neural machine translation. However, in-domain gains from adaptation come at the expense of general-domain performance. In this work, we interpret the drop in general-domain performance as catastrophic forgetting of general-domain knowledge. To mitigate it, we adapt Elastic Weight Consolidation (EWC)—a machine learning method for learning a new task without forgetting previous tasks. Our method retains the majority of general-domain performance lost in continued training without degrading indomain performance, outperforming the previous state-of-the-art. We also explore the full range of general-domain performance available when some in-domain degradation is acceptable.

U2 - 10.18653/v1/n19-1209

DO - 10.18653/v1/n19-1209

M3 - Conference contribution

VL - 1

SP - 2062

EP - 2068

BT - Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)

A2 - Burstein, Jill

A2 - Doran, Christy

A2 - Solorio, Thamar

PB - Association for Computational Linguistics

CY - Minneapolis, Minnesota

T2 - 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics

Y2 - 2 June 2019 through 7 June 2019

ER -

Thompson B, Gwinnup J, Khayrallah H, Duh K, Koehn P. Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation. In Burstein J, Doran C, Solorio T, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Vol. 1. Minneapolis, Minnesota: Association for Computational Linguistics. 2019. p. 2062–2068 https://doi.org/10.18653/v1/n19-1209

Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation

Abstract

Conference

Access to Document

Fingerprint

Cite this