Climbing Mont BLEU: The Strange World of Reachable High-BLEU Translations

Aaron Smith; Christian Hardmeier; Joerg Tiedemann

Climbing Mont BLEU: The Strange World of Reachable High-BLEU Translations

Aaron Smith, Christian Hardmeier, Joerg Tiedemann

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

We present a method for finding oracle BLEU translations in phrase-based statistical machine translation using exact document-level scores. Experiments are presented where the BLEU score of a candidate translation is directly optimised in order to examine the properties of reachable translations with very high BLEU scores. This is achieved by running the document-level decoder Docent in BLEU-decoding mode, where proposed changes to the translation of a document are only accepted if they increase BLEU. The results confirm that the reference translation cannot in most cases be reached by the decoder, which is limited by the set of phrases in the phrase table, and demonstrate that high-BLEU translations are often of poor quality.

Original language	English
Title of host publication	Proceedings of the 19th Annual Conference of the European Association for Machine Translation
Publisher	Association for Computational Linguistics
Pages	269-281
Number of pages	13
Publication status	Published - 1 Jun 2016
Event	19th annual conference of the European Association for Machine Translation (EAMT) - Riga, Latvia Duration: 30 May 2016 → 1 Jun 2016 Conference number: 19 http://eamt2016.tilde.com/

Conference

Conference	19th annual conference of the European Association for Machine Translation (EAMT)
Abbreviated title	EAMT 2016
Country/Territory	Latvia
City	Riga
Period	30/05/16 → 1/06/16
Internet address	http://eamt2016.tilde.com/

Keywords / Materials (for Non-textual outputs)

Statistical machine translation
oracle decoding
BLEU
Docent

Access to Document

Climbing Mount BLEU_SMITH_DOA_22042016_VOR_CC-BYFinal published version, 170 KBLicence: Creative Commons: Attribution (CC-BY)

https://www.aclweb.org/anthology/W16-3414Licence: Creative Commons: Attribution (CC-BY)

Cite this

@inproceedings{b2babe6d181f411d9f573f11b400df37,

title = "Climbing Mont BLEU: The Strange World of Reachable High-BLEU Translations",

abstract = "We present a method for finding oracle BLEU translations in phrase-based statistical machine translation using exact document-level scores. Experiments are presented where the BLEU score of a candidate translation is directly optimised in order to examine the properties of reachable translations with very high BLEU scores. This is achieved by running the document-level decoder Docent in BLEU-decoding mode, where proposed changes to the translation of a document are only accepted if they increase BLEU. The results confirm that the reference translation cannot in most cases be reached by the decoder, which is limited by the set of phrases in the phrase table, and demonstrate that high-BLEU translations are often of poor quality.",

keywords = "Statistical machine translation, oracle decoding, BLEU, Docent",

author = "Aaron Smith and Christian Hardmeier and Joerg Tiedemann",

year = "2016",

month = jun,

day = "1",

language = "English",

pages = "269--281",

booktitle = "Proceedings of the 19th Annual Conference of the European Association for Machine Translation",

publisher = "Association for Computational Linguistics",

note = "19th annual conference of the European Association for Machine Translation (EAMT), EAMT 2016 ; Conference date: 30-05-2016 Through 01-06-2016",

url = "http://eamt2016.tilde.com/",

}

Smith, A, Hardmeier, C & Tiedemann, J 2016, Climbing Mont BLEU: The Strange World of Reachable High-BLEU Translations. in Proceedings of the 19th Annual Conference of the European Association for Machine Translation. Association for Computational Linguistics, pp. 269-281, 19th annual conference of the European Association for Machine Translation (EAMT), Riga, Latvia, 30/05/16. <https://www.aclweb.org/anthology/W16-3414>

Climbing Mont BLEU: The Strange World of Reachable High-BLEU Translations. / Smith, Aaron; Hardmeier, Christian; Tiedemann, Joerg.
Proceedings of the 19th Annual Conference of the European Association for Machine Translation. Association for Computational Linguistics, 2016. p. 269-281.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Climbing Mont BLEU: The Strange World of Reachable High-BLEU Translations

AU - Smith, Aaron

AU - Hardmeier, Christian

AU - Tiedemann, Joerg

N1 - Conference code: 19

PY - 2016/6/1

Y1 - 2016/6/1

N2 - We present a method for finding oracle BLEU translations in phrase-based statistical machine translation using exact document-level scores. Experiments are presented where the BLEU score of a candidate translation is directly optimised in order to examine the properties of reachable translations with very high BLEU scores. This is achieved by running the document-level decoder Docent in BLEU-decoding mode, where proposed changes to the translation of a document are only accepted if they increase BLEU. The results confirm that the reference translation cannot in most cases be reached by the decoder, which is limited by the set of phrases in the phrase table, and demonstrate that high-BLEU translations are often of poor quality.

AB - We present a method for finding oracle BLEU translations in phrase-based statistical machine translation using exact document-level scores. Experiments are presented where the BLEU score of a candidate translation is directly optimised in order to examine the properties of reachable translations with very high BLEU scores. This is achieved by running the document-level decoder Docent in BLEU-decoding mode, where proposed changes to the translation of a document are only accepted if they increase BLEU. The results confirm that the reference translation cannot in most cases be reached by the decoder, which is limited by the set of phrases in the phrase table, and demonstrate that high-BLEU translations are often of poor quality.

KW - Statistical machine translation

KW - oracle decoding

KW - BLEU

KW - Docent

M3 - Conference contribution

SP - 269

EP - 281

BT - Proceedings of the 19th Annual Conference of the European Association for Machine Translation

PB - Association for Computational Linguistics

T2 - 19th annual conference of the European Association for Machine Translation (EAMT)

Y2 - 30 May 2016 through 1 June 2016

ER -

Climbing Mont BLEU: The Strange World of Reachable High-BLEU Translations

Abstract / Description of output

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this