Statistical Significance Tests for Machine Translation Evaluation

Philipp Koehn

Statistical Significance Tests for Machine Translation Evaluation

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

If two translation systems differ differ in performance on a test set, can we trust that this indicates a difference in true system quality? To answer this question, we describe bootstrap resampling methods to compute statistical significance of test results, and validate them on the concrete example of the BLEU score. Even for small test sizes of only 300 sentences, our methods may give us assurances that test result differences are real.

Original language	English
Title of host publication	Proceedings of EMNLP 2004
Editors	Dekang Lin, Dekai Wu
Place of Publication	Barcelona, Spain
Publisher	Association for Computational Linguistics
Pages	388-395
Number of pages	8
Publication status	Published - 1 Jul 2004

Access to Document

http://www.aclweb.org/anthology/W/W04/W04-3250.pdf

Cite this

@inproceedings{29ded0fe03b9496b831518596c237019,

title = "Statistical Significance Tests for Machine Translation Evaluation",

abstract = "If two translation systems differ differ in performance on a test set, can we trust that this indicates a difference in true system quality? To answer this question, we describe bootstrap resampling methods to compute statistical significance of test results, and validate them on the concrete example of the BLEU score. Even for small test sizes of only 300 sentences, our methods may give us assurances that test result differences are real.",

author = "Philipp Koehn",

year = "2004",

month = jul,

day = "1",

language = "English",

pages = "388--395",

editor = "Dekang Lin and Dekai Wu",

booktitle = "Proceedings of EMNLP 2004",

publisher = "Association for Computational Linguistics",

}

TY - GEN

T1 - Statistical Significance Tests for Machine Translation Evaluation

AU - Koehn, Philipp

PY - 2004/7/1

Y1 - 2004/7/1

N2 - If two translation systems differ differ in performance on a test set, can we trust that this indicates a difference in true system quality? To answer this question, we describe bootstrap resampling methods to compute statistical significance of test results, and validate them on the concrete example of the BLEU score. Even for small test sizes of only 300 sentences, our methods may give us assurances that test result differences are real.

AB - If two translation systems differ differ in performance on a test set, can we trust that this indicates a difference in true system quality? To answer this question, we describe bootstrap resampling methods to compute statistical significance of test results, and validate them on the concrete example of the BLEU score. Even for small test sizes of only 300 sentences, our methods may give us assurances that test result differences are real.

M3 - Conference contribution

SP - 388

EP - 395

BT - Proceedings of EMNLP 2004

A2 - Lin, Dekang

A2 - Wu, Dekai

PB - Association for Computational Linguistics

CY - Barcelona, Spain

ER -

Statistical Significance Tests for Machine Translation Evaluation

Abstract

Access to Document

Fingerprint

Cite this