How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Analysing translation quality in regards to specific linguistic phenomena has historically been difficult and time-consuming. Neural machine translation has the attractive property that it can produce scores for arbitrary translations, and we propose a novel method to assess how well NMT systems model specific linguistic phenomena such as agreement over long distances, the production of novel words, and the faithful translation of polarity. The core idea is that we measure whether a reference translation is more probable under a NMT model than a contrastive translation which introduces a specific type of error. We present LingEval971, a large-scale data set of 97 000 contrastive translation pairs based on the WMT English→German translation task, with errors automatically created with simple rules. We report results for a number of systems, and find that recently introduced character-level NMT systems perform better at transliteration than models with byte-pair encoding (BPE) segmentation, but perform more poorly at morphosyntactic agreement, and translating discontiguous units of meaning.
Original languageEnglish
Title of host publicationProceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers
Place of PublicationValencia, Spain
PublisherAssociation for Computational Linguistics (ACL)
Pages376-382
Number of pages7
ISBN (Print)978-1-945626-34-0
Publication statusPublished - 7 Apr 2017
Event15th EACL 2017 Software Demonstrations - Valencia, Spain
Duration: 3 Apr 20177 Apr 2017
http://eacl2017.org/
http://eacl2017.org/index.php

Conference

Conference15th EACL 2017 Software Demonstrations
Abbreviated titleEACL 2017
Country/TerritorySpain
CityValencia
Period3/04/177/04/17
Internet address

Fingerprint

Dive into the research topics of 'How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs'. Together they form a unique fingerprint.

Cite this