A Pronoun Test Suite Evaluation of the English--German MT Systems at WMT 2018

Liane Guillou, Christian Hardmeier, Ekaterina Lapshinova-Koltunski, Sharid Loáiciga

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We evaluate the output of 16 English-to-German MT systems with respect to the translation of pronouns in the context of the WMT 2018 competition. We work with a test suite specifically designed to assess system quality in various fine-grained categories known to be problematic. The main evaluation scores come from a semi-automatic process, combining automatic reference matching with extensive manual annotation of uncertain cases. We find that current NMT systems are good at translating pronouns with intra-sentential reference, but the inter-sentential cases remain difficult. NMT systems are also good at the translation of event pronouns, unlike systems from the phrase-based SMT paradigm. No single system performs best at translating all types of anaphoric pronouns, suggesting unexplained random effects influencing the translation of pronouns with NMT.
Original languageEnglish
Title of host publicationProceedings of the Third Conference on Machine Translation: Shared Task Papers
Place of PublicationBelgium, Brussels
PublisherAssociation for Computational Linguistics
Number of pages8
ISBN (Electronic)978-1-948087-81-0
Publication statusPublished - 1 Nov 2018
EventEMNLP 2018 Third Conference on Machine Translation (WMT18) - Brussels, Belgium
Duration: 31 Oct 20181 Nov 2018


WorkshopEMNLP 2018 Third Conference on Machine Translation (WMT18)
Abbreviated titleWMT18
Internet address


Dive into the research topics of 'A Pronoun Test Suite Evaluation of the English--German MT Systems at WMT 2018'. Together they form a unique fingerprint.

Cite this