Hierarchical Permutation Complexity for Word Order Evaluation

Milos Stanojevic, Khalil Sima'an

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Existing approaches for evaluating word order in machine translation work with metrics computed directly over a permutation of word positions in system output relative to a reference translation. However, every permutation factorizes into a permutation tree (PET) built of primal permutations, i.e., atomic units that do not factorize any further. In this paper we explore the idea that permutations factorizing into (on average) shorter primal permutations should represent simpler ordering as well. Consequently, we contribute Permutation Complexity, a class of metrics over PETs and their extension to forests, and define tight metrics, a sub-class of metrics implementing this idea. Subsequently we define example tight metrics and empirically test them in word order evaluation. Experiments on the WMT13 data sets for ten language pairs show that a tight metric is more often than not better than the baselines.
Original languageEnglish
Title of host publicationProceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
Place of PublicationOsaka, Japan
PublisherThe COLING 2016 Organizing Committee
Number of pages10
Publication statusPublished - 30 Nov 2016
Event26th International Conference on Computational Linguistics - Osaka, Japan
Duration: 11 Dec 201616 Dec 2016


Conference26th International Conference on Computational Linguistics
Abbreviated titleCOLING 2016
Internet address


Dive into the research topics of 'Hierarchical Permutation Complexity for Word Order Evaluation'. Together they form a unique fingerprint.

Cite this