Results of the WMT15 Metrics Shared Task

Milos Stanojevic; Amir  Kamran; Philipp Koehn; Ondrej Bojar

Results of the WMT15 Metrics Shared Task

Milos Stanojevic, Amir Kamran, Philipp Koehn, Ondrej Bojar

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper presents the results of the WMT15 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT15 Shared Translation Task. We collected scores of 46 metrics from 11 research groups. In addition to that, we computed scores of 7 standard metrics(BLEU, SentBLEU, NIST, WER, PER,TER and CDER) as baselines. The collected scores were evaluated in terms of system level correlation (how well each metric’s scores correlate with WMT15 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence).

Original language	English
Title of host publication	Proceedings of the Tenth Workshop on Statistical Machine Translation, 2015
Place of Publication	Lisbon, Portugal
Publisher	Association for Computational Linguistics
Pages	256-273
Number of pages	18
Publication status	Published - 2015
Event	Tenth Workshop on Statistical Machine Translation - Lisbon, Portugal Duration: 17 Sep 2015 → 18 Sep 2015 http://www.statmt.org/wmt15/

Conference

Conference	Tenth Workshop on Statistical Machine Translation
Abbreviated title	EMNLP 2015
Country/Territory	Portugal
City	Lisbon
Period	17/09/15 → 18/09/15
Internet address	http://www.statmt.org/wmt15/

Access to Document

W15-3031Final published version, 583 KB

http://www.statmt.org/wmt15/papers.html

2 Finished

QT21: Quality Translation 21
Koehn, P., Haddow, B. & Huck, M.
EU government bodies
1/02/15 → 31/01/18
Project: Research
HimL: Health in my Language
Haddow, B., Birch-Mayne, A. & Webber, B.
EU government bodies
1/02/15 → 31/01/18
Project: Research

Cite this

@inproceedings{baba589b9d254e488f4d0bff25d7b274,

title = "Results of the WMT15 Metrics Shared Task",

abstract = "This paper presents the results of the WMT15 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT15 Shared Translation Task. We collected scores of 46 metrics from 11 research groups. In addition to that, we computed scores of 7 standard metrics(BLEU, SentBLEU, NIST, WER, PER,TER and CDER) as baselines. The collected scores were evaluated in terms of system level correlation (how well each metric{\textquoteright}s scores correlate with WMT15 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence).",

author = "Milos Stanojevic and Amir Kamran and Philipp Koehn and Ondrej Bojar",

year = "2015",

language = "English",

pages = "256--273",

booktitle = "Proceedings of the Tenth Workshop on Statistical Machine Translation, 2015",

publisher = "Association for Computational Linguistics",

note = "Tenth Workshop on Statistical Machine Translation, EMNLP 2015 ; Conference date: 17-09-2015 Through 18-09-2015",

url = "http://www.statmt.org/wmt15/",

}

TY - GEN

T1 - Results of the WMT15 Metrics Shared Task

AU - Stanojevic, Milos

AU - Kamran, Amir

AU - Koehn, Philipp

AU - Bojar, Ondrej

PY - 2015

Y1 - 2015

N2 - This paper presents the results of the WMT15 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT15 Shared Translation Task. We collected scores of 46 metrics from 11 research groups. In addition to that, we computed scores of 7 standard metrics(BLEU, SentBLEU, NIST, WER, PER,TER and CDER) as baselines. The collected scores were evaluated in terms of system level correlation (how well each metric’s scores correlate with WMT15 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence).

AB - This paper presents the results of the WMT15 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in the WMT15 Shared Translation Task. We collected scores of 46 metrics from 11 research groups. In addition to that, we computed scores of 7 standard metrics(BLEU, SentBLEU, NIST, WER, PER,TER and CDER) as baselines. The collected scores were evaluated in terms of system level correlation (how well each metric’s scores correlate with WMT15 official manual ranking of systems) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence).

M3 - Conference contribution

SP - 256

EP - 273

BT - Proceedings of the Tenth Workshop on Statistical Machine Translation, 2015

PB - Association for Computational Linguistics

CY - Lisbon, Portugal

T2 - Tenth Workshop on Statistical Machine Translation

Y2 - 17 September 2015 through 18 September 2015

ER -

Results of the WMT15 Metrics Shared Task

Abstract

Conference

Access to Document

Projects

QT21: Quality Translation 21

HimL: Health in my Language

Cite this