The Word Sense Disambiguation Test Suite at WMT18

Annette Rios, Mathias Müller, Rico Sennrich

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present a task to measure an MT system’s capability to translate ambiguous words with their correct sense according to the given context. The task is based on the German–English Word Sense Disambiguation (WSD) test set ContraWSD (Rios Gonzales et al., 2017), but it has been filtered to reduce noise, and the evaluation has been adapted to assess MT output directly rather than scoring existing translations. We evaluate all German–English submissions to the WMT’18 shared translation task, plus a number of submissions from previous years, and find that performance on the task has markedly improved compared to the 2016 WMT submissions (81%!93% accuracy on the WSD task). We also find that the unsupervised submissions to the task have a low WSD capability, and predominantly translate ambiguous source words with the same sense.
Original languageEnglish
Title of host publicationEMNLP 2018 THIRD CONFERENCE ON MACHINE TRANSLATION (WMT18)
Place of PublicationBrussels, Belgium
PublisherAssociation for Computational Linguistics
Pages588-596
Number of pages9
DOIs
Publication statusPublished - Oct 2018
EventEMNLP 2018 Third Conference on Machine Translation (WMT18) - Brussels, Belgium
Duration: 31 Oct 20181 Nov 2018
http://www.statmt.org/wmt18/

Workshop

WorkshopEMNLP 2018 Third Conference on Machine Translation (WMT18)
Abbreviated titleWMT18
CountryBelgium
CityBrussels
Period31/10/181/11/18
Internet address

Cite this