Evaluating Discourse Phenomena in Neural Machine Translation

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

For machine translation to tackle discourse phenomena, models must have access to extrasentential linguistic context. There has been recent interest in modelling context in neural machine translation (NMT), but models have been principally evaluated with standard automatic metrics, poorly adapted to evaluating discourse phenomena. In this article, we present hand-crafted, discourse test sets, designed to test the models’ ability to exploit previous source and target sentences. We investigate the performance of recently proposed multi-encoder NMT models trained on subtitles for English to French. We also explore a novel way of exploiting context from the previous sentence. Despite gains using BLEU, multi-encoder models give limited improvement in the handling of discourse phenomena: 50% accuracy on our coreference test set and 53.5% for coherence/cohesion (compared to a non-contextual baseline of 50%). A simple strategy of decoding the concatenation of the previous and current sentence leads to good performance, and our novel strategy of multiencoding and decoding of two sentences leads to the best performance (72.5% for coreference and 57% for coherence/cohesion), highlighting the importance of target-side context.
Original languageEnglish
Title of host publicationProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Place of PublicationNew Orleans, Louisiana
PublisherAssociation for Computational Linguistics (ACL)
Pages1304-1313
Number of pages10
ISBN (Electronic)978-1-948087-27-8
DOIs
Publication statusPublished - 6 Jun 2018
Event16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Hyatt Regency New Orleans Hotel, New Orleans, United States
Duration: 1 Jun 20186 Jun 2018
http://naacl2018.org/

Conference

Conference16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Abbreviated titleNAACL HLT 2018
CountryUnited States
CityNew Orleans
Period1/06/186/06/18
Internet address

Fingerprint

Dive into the research topics of 'Evaluating Discourse Phenomena in Neural Machine Translation'. Together they form a unique fingerprint.

Cite this