Edinburgh Research Explorer

How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Related Edinburgh Organisations

Open Access permissions

Open

Documents

http://www.aclweb.org/anthology/E/E17/E17-2060.pdf
https://aclweb.org/anthology/papers/E/E17/E17-2060/
Original languageEnglish
Title of host publicationProceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers
Place of PublicationValencia, Spain
PublisherAssociation for Computational Linguistics (ACL)
Pages376-382
Number of pages7
ISBN (Print)978-1-945626-34-0
Publication statusPublished - 7 Apr 2017
Event15th EACL 2017 Software Demonstrations - Valencia, Spain
Duration: 3 Apr 20177 Apr 2017
http://eacl2017.org/
http://eacl2017.org/index.php

Conference

Conference15th EACL 2017 Software Demonstrations
Abbreviated titleEACL 2017
CountrySpain
CityValencia
Period3/04/177/04/17
Internet address

Abstract

Analysing translation quality in regards to specific linguistic phenomena has historically been difficult and time-consuming. Neural machine translation has the attractive property that it can produce scores for arbitrary translations, and we propose a novel method to assess how well NMT systems model specific linguistic phenomena such as agreement over long distances, the production of novel words, and the faithful translation of polarity. The core idea is that we measure whether a reference translation is more probable under a NMT model than a contrastive translation which introduces a specific type of error. We present LingEval971, a large-scale data set of 97 000 contrastive translation pairs based on the WMT English→German translation task, with errors automatically created with simple rules. We report results for a number of systems, and find that recently introduced character-level NMT systems perform better at transliteration than models with byte-pair encoding (BPE) segmentation, but perform more poorly at morphosyntactic agreement, and translating discontiguous units of meaning.

Event

15th EACL 2017 Software Demonstrations

3/04/177/04/17

Valencia, Spain

Event: Conference

Download statistics

No data available

ID: 31460655