The Word Sense Disambiguation Test Suite at WMT18

Annette Rios; Mathias Müller; Rico Sennrich

doi:10.18653/v1/W18-6437

The Word Sense Disambiguation Test Suite at WMT18

Annette Rios, Mathias Müller, Rico Sennrich

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We present a task to measure an MT system’s capability to translate ambiguous words with their correct sense according to the given context. The task is based on the German–English Word Sense Disambiguation (WSD) test set ContraWSD (Rios Gonzales et al., 2017), but it has been filtered to reduce noise, and the evaluation has been adapted to assess MT output directly rather than scoring existing translations. We evaluate all German–English submissions to the WMT’18 shared translation task, plus a number of submissions from previous years, and find that performance on the task has markedly improved compared to the 2016 WMT submissions (81%!93% accuracy on the WSD task). We also find that the unsupervised submissions to the task have a low WSD capability, and predominantly translate ambiguous source words with the same sense.

Original language	English
Title of host publication	EMNLP 2018 THIRD CONFERENCE ON MACHINE TRANSLATION (WMT18)
Place of Publication	Brussels, Belgium
Publisher	Association for Computational Linguistics
Pages	588-596
Number of pages	9
DOIs	https://doi.org/10.18653/v1/W18-6437
Publication status	Published - Oct 2018
Event	EMNLP 2018 Third Conference on Machine Translation (WMT18) - Brussels, Belgium Duration: 31 Oct 2018 → 1 Nov 2018 http://www.statmt.org/wmt18/

Workshop

Workshop	EMNLP 2018 Third Conference on Machine Translation (WMT18)
Abbreviated title	WMT18
Country/Territory	Belgium
City	Brussels
Period	31/10/18 → 1/11/18
Internet address	http://www.statmt.org/wmt18/

Access to Document

10.18653/v1/W18-6437Licence: Creative Commons: Attribution (CC-BY)

the word sense disambiguation test suite at WMT18Accepted author manuscript, 141 KB
the word sense disambiguation test suite at WMT18Final published version, 141 KBLicence: Creative Commons: Attribution (CC-BY)

https://www.aclweb.org/anthology/W18-6437Licence: Creative Commons: Attribution (CC-BY)

Cite this

@inproceedings{d8d03f9b9c17428583f9adb41cb609fe,

title = "The Word Sense Disambiguation Test Suite at WMT18",

abstract = "We present a task to measure an MT system{\textquoteright}s capability to translate ambiguous words with their correct sense according to the given context. The task is based on the German–English Word Sense Disambiguation (WSD) test set ContraWSD (Rios Gonzales et al., 2017), but it has been filtered to reduce noise, and the evaluation has been adapted to assess MT output directly rather than scoring existing translations. We evaluate all German–English submissions to the WMT{\textquoteright}18 shared translation task, plus a number of submissions from previous years, and find that performance on the task has markedly improved compared to the 2016 WMT submissions (81%!93% accuracy on the WSD task). We also find that the unsupervised submissions to the task have a low WSD capability, and predominantly translate ambiguous source words with the same sense.",

author = "Annette Rios and Mathias M{\"u}ller and Rico Sennrich",

year = "2018",

month = oct,

doi = "10.18653/v1/W18-6437",

language = "English",

pages = "588--596",

booktitle = "EMNLP 2018 THIRD CONFERENCE ON MACHINE TRANSLATION (WMT18)",

publisher = "Association for Computational Linguistics",

note = "EMNLP 2018 Third Conference on Machine Translation (WMT18), WMT18 ; Conference date: 31-10-2018 Through 01-11-2018",

url = "http://www.statmt.org/wmt18/",

}

TY - GEN

T1 - The Word Sense Disambiguation Test Suite at WMT18

AU - Rios, Annette

AU - Müller, Mathias

AU - Sennrich, Rico

PY - 2018/10

Y1 - 2018/10

N2 - We present a task to measure an MT system’s capability to translate ambiguous words with their correct sense according to the given context. The task is based on the German–English Word Sense Disambiguation (WSD) test set ContraWSD (Rios Gonzales et al., 2017), but it has been filtered to reduce noise, and the evaluation has been adapted to assess MT output directly rather than scoring existing translations. We evaluate all German–English submissions to the WMT’18 shared translation task, plus a number of submissions from previous years, and find that performance on the task has markedly improved compared to the 2016 WMT submissions (81%!93% accuracy on the WSD task). We also find that the unsupervised submissions to the task have a low WSD capability, and predominantly translate ambiguous source words with the same sense.

AB - We present a task to measure an MT system’s capability to translate ambiguous words with their correct sense according to the given context. The task is based on the German–English Word Sense Disambiguation (WSD) test set ContraWSD (Rios Gonzales et al., 2017), but it has been filtered to reduce noise, and the evaluation has been adapted to assess MT output directly rather than scoring existing translations. We evaluate all German–English submissions to the WMT’18 shared translation task, plus a number of submissions from previous years, and find that performance on the task has markedly improved compared to the 2016 WMT submissions (81%!93% accuracy on the WSD task). We also find that the unsupervised submissions to the task have a low WSD capability, and predominantly translate ambiguous source words with the same sense.

U2 - 10.18653/v1/W18-6437

DO - 10.18653/v1/W18-6437

M3 - Conference contribution

SP - 588

EP - 596

BT - EMNLP 2018 THIRD CONFERENCE ON MACHINE TRANSLATION (WMT18)

PB - Association for Computational Linguistics

CY - Brussels, Belgium

T2 - EMNLP 2018 Third Conference on Machine Translation (WMT18)

Y2 - 31 October 2018 through 1 November 2018

ER -

The Word Sense Disambiguation Test Suite at WMT18

Abstract

Workshop

Access to Document

Fingerprint

Cite this