In Neural Machine Translation, What Does Transfer Learning Transfer?

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Transfer learning improves quality for low-resource machine translation, but it is unclear what exactly it transfers. We perform several ablation studies that limit information transfer, then measure the quality impact across three language pairs to gain a black-box understanding of transfer learning. Word embeddings play an important role in transfer learning, particularly if they are properly aligned. Although transfer learning can be performed without embeddings, results are sub-optimal. In contrast, transferring only the embeddings but nothing else yields catastrophic results. We then investigate diagonal alignments with auto-encoders over real languages and randomly generated sequences, finding even randomly generated sequences as parents yield noticeable but smaller gains. Finally, transfer learning can eliminate the need for a warm-up phase when training transformer models in high resource language pairs.
Original languageEnglish
Title of host publicationProceedings of the 58th Annual Meeting of the Association for Computational Linguistics
PublisherAssociation for Computational Linguistics (ACL)
Pages7701–7710
Number of pages10
ISBN (Electronic)978-1-952148-25-5
DOIs
Publication statusPublished - 10 Jul 2020
Event2020 Annual Conference of the Association for Computational Linguistics - Hyatt Regency Seattle, Virtual conference, United States
Duration: 5 Jul 202010 Jul 2020
Conference number: 58
https://acl2020.org/

Conference

Conference2020 Annual Conference of the Association for Computational Linguistics
Abbreviated titleACL 2020
Country/TerritoryUnited States
CityVirtual conference
Period5/07/2010/07/20
Internet address

Fingerprint

Dive into the research topics of 'In Neural Machine Translation, What Does Transfer Learning Transfer?'. Together they form a unique fingerprint.

Cite this