A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators

Sheila Castilho, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Yota Georgakopoulou, Pintu Lohar, Andy Way, Antonio Miceli Barone, Maria Gialama

Research output: Chapter in Book/Report/Conference proceedingConference contribution


This paper reports on a comparative evaluation of phrase-based statistical machine translation (PBSMT) and neural machine translation (NMT) for four language pairs, using the PET interface to compare educational domain output from both systems using a variety of metrics, including automatic evaluation as well as human rankings of adequacy and fluency, error-type markup, and post-editing (technical and temporal) effort, performed by professional translators.
Our results show a preference for NMT in side-by-side ranking for all language pairs, texts, and segment lengths. In addition, perceived fluency is improved and annotated errors are fewer in the NMT output. Results are mixed for perceived adequacy and for errors of omission, addition, and mistranslation. Despite far fewer segments requiring post-editing, document-level post-editing performance was not found to have significantly improved in NMT compared to
PBSMT. This evaluation was conducted as part of the TraMOOC project, which aims to create a replicable semi-automated methodology for high-quality machine translation of educational data.
Original languageEnglish
Title of host publicationProceedings of MT Summit XVI, vol.1: Research Track
Place of PublicationNagoya, Japan
Number of pages16
Publication statusPublished - 11 Sep 2017
EventMachine Translation Summit XVI 2017 - Nagoya University, Nagoya, Japan
Duration: 18 Sep 201722 Sep 2017


ConferenceMachine Translation Summit XVI 2017
Abbreviated titleMT XVI 2017
Internet address

Cite this