Abstract
One of the major challenges facing statistical machine translation is how to model differences in word order between languages. Although
a great deal of research has focussed on this problem, progress is hampered by the lack of reliable metrics. Most current metrics
are based on matching lexical items in the translation and the reference, and their ability to measure the quality of word order has not been demonstrated. This paper presents a novel metric, the LRscore, which explicitly measures the quality of word order by using permutation distance metrics. We show that the metric is more consistent with human
judgements than other metrics, including the BLEU score. We also show that the LRscore can successfully be used as the objective function when training translation model parameters. Training with the LRscore leads to output which is preferred by humans. Moreover, the translations incur no penalty in terms of BLEU scores.
a great deal of research has focussed on this problem, progress is hampered by the lack of reliable metrics. Most current metrics
are based on matching lexical items in the translation and the reference, and their ability to measure the quality of word order has not been demonstrated. This paper presents a novel metric, the LRscore, which explicitly measures the quality of word order by using permutation distance metrics. We show that the metric is more consistent with human
judgements than other metrics, including the BLEU score. We also show that the LRscore can successfully be used as the objective function when training translation model parameters. Training with the LRscore leads to output which is preferred by humans. Moreover, the translations incur no penalty in terms of BLEU scores.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies |
| Publisher | Association for Computational Linguistics |
| Pages | 1027-1035 |
| Number of pages | 9 |
| ISBN (Print) | 978-1-932432-87-9 |
| Publication status | Published - 2011 |