Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In Neural Machine Translation (and, more generally, conditional language modeling), the generation of a target token is influenced by two types of context: the source and the prefix of the target sequence. While many attempts to understand the internal workings of NMT models have been made, none of them explicitly evaluates relative source and target contributions to a generation decision. We argue that this relative contribution can be evaluated by adopting a variant of Layerwise Relevance Propagation (LRP). Its underlying ‘conservation principle’ makes relevance propagation unique: differently from other methods, ite valuates not an abstract quantity reflecting token importance, but the proportion of each token’s influence. We extend LRP to the Transformer and conduct an analysis of NMT models which explicitly evaluates the source and target relative contributions to the generation process. We analyze changes in these contributions when conditioning on different types of prefixes, when varying the training objective or the amount of training data, and during the training process. We find that models trained with more data tend to rely on source information more and to have more sharp token contributions; the training process is non-monotonic with several stages of different nature.
Original languageEnglish
Title of host publicationProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Place of PublicationOnline
PublisherAssociation for Computational Linguistics
Pages1126-1140
Number of pages15
ISBN (Electronic)978-1-954085-52-7
Publication statusPublished - 1 Aug 2021
EventThe Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing - Bangkok, Thailand
Duration: 1 Aug 20216 Aug 2021
https://2021.aclweb.org/

Conference

ConferenceThe Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing
Abbreviated titleACL-IJCNLP 2021
Country/TerritoryThailand
CityBangkok
Period1/08/216/08/21
Internet address

Fingerprint

Dive into the research topics of 'Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation'. Together they form a unique fingerprint.

Cite this