Evaluating MT systems with BEER

Milos Stanojevic, Khalil Sima'an

Research output: Contribution to journalArticlepeer-review


We present BEER, an open source implementation of a machine translation evaluation metric. BEER is a metric trained for high correlation with human ranking by using learning-to-rank training methods. For evaluation of lexical accuracy it uses sub-word units (character n-grams) while for measuring word order it uses hierarchical representations based on PETs (permutation trees). During the last WMT metrics tasks, BEER has shown high correlation with human judgments both on the sentence and the corpus levels. In this paper we will show how BEER can be used for (i) full evaluation of MT output,(ii) isolated evaluation of word order and (iii) tuning MT systems.
Original languageEnglish
Pages (from-to)17-26
Number of pages10
JournalPrague Bulletin of Mathematical Linguistics
Issue number1
Publication statusPublished - 7 Sep 2015


Dive into the research topics of 'Evaluating MT systems with BEER'. Together they form a unique fingerprint.

Cite this