A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators

Sheila Castilho; Joss Moorkens; Federico Gaspari; Rico Sennrich; Vilelmini Sosoni; Yota Georgakopoulou; Pintu Lohar; Andy Way; Antonio Miceli Barone; Maria Gialama

A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators

Sheila Castilho, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Yota Georgakopoulou, Pintu Lohar, Andy Way, Antonio Miceli Barone, Maria Gialama

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper reports on a comparative evaluation of phrase-based statistical machine translation (PBSMT) and neural machine translation (NMT) for four language pairs, using the PET interface to compare educational domain output from both systems using a variety of metrics, including automatic evaluation as well as human rankings of adequacy and fluency, error-type markup, and post-editing (technical and temporal) effort, performed by professional translators.
Our results show a preference for NMT in side-by-side ranking for all language pairs, texts, and segment lengths. In addition, perceived fluency is improved and annotated errors are fewer in the NMT output. Results are mixed for perceived adequacy and for errors of omission, addition, and mistranslation. Despite far fewer segments requiring post-editing, document-level post-editing performance was not found to have significantly improved in NMT compared to
PBSMT. This evaluation was conducted as part of the TraMOOC project, which aims to create a replicable semi-automated methodology for high-quality machine translation of educational data.

Original language	English
Title of host publication	Proceedings of MT Summit XVI, vol.1: Research Track
Place of Publication	Nagoya, Japan
Pages	116-131
Number of pages	16
Publication status	Published - 11 Sep 2017
Event	Machine Translation Summit XVI 2017 - Nagoya University, Nagoya, Japan Duration: 18 Sep 2017 → 22 Sep 2017 http://www.eamt.org/news/news_cfp_mts2017.php

Conference

Conference	Machine Translation Summit XVI 2017
Abbreviated title	MT XVI 2017
Country/Territory	Japan
City	Nagoya
Period	18/09/17 → 22/09/17
Internet address	http://www.eamt.org/news/news_cfp_mts2017.php

Access to Document

Castilho et al 2017_MTSummit-1Accepted author manuscript, 178 KB

http://aamt.info/app-def/S-102/mtsummit/2017/wp-content/uploads/sites/2/2017/09/MTSummitXVI_ResearchTrack.pdfLicence: Creative Commons: Attribution No Derivatives (CC-BY-ND)

Translation for Massive Open Online Courses- TraMooc
Koehn, P. & Birch-Mayne, A.
EU government bodies
1/02/15 → 31/01/18
Project: Research

Cite this

Castilho, S., Moorkens, J., Gaspari, F., Sennrich, R., Sosoni, V., Georgakopoulou, Y., Lohar, P., Way, A., Miceli Barone, A., & Gialama, M. (2017). A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators. In Proceedings of MT Summit XVI, vol.1: Research Track (pp. 116-131). http://aamt.info/app-def/S-102/mtsummit/2017/wp-content/uploads/sites/2/2017/09/MTSummitXVI_ResearchTrack.pdf

@inproceedings{9f92fc62f43844f1b8d35d31cfa6d807,

title = "A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators",

abstract = "This paper reports on a comparative evaluation of phrase-based statistical machine translation (PBSMT) and neural machine translation (NMT) for four language pairs, using the PET interface to compare educational domain output from both systems using a variety of metrics, including automatic evaluation as well as human rankings of adequacy and fluency, error-type markup, and post-editing (technical and temporal) effort, performed by professional translators.Our results show a preference for NMT in side-by-side ranking for all language pairs, texts, and segment lengths. In addition, perceived fluency is improved and annotated errors are fewer in the NMT output. Results are mixed for perceived adequacy and for errors of omission, addition, and mistranslation. Despite far fewer segments requiring post-editing, document-level post-editing performance was not found to have significantly improved in NMT compared toPBSMT. This evaluation was conducted as part of the TraMOOC project, which aims to create a replicable semi-automated methodology for high-quality machine translation of educational data.",

author = "Sheila Castilho and Joss Moorkens and Federico Gaspari and Rico Sennrich and Vilelmini Sosoni and Yota Georgakopoulou and Pintu Lohar and Andy Way and {Miceli Barone}, Antonio and Maria Gialama",

year = "2017",

month = sep,

day = "11",

language = "English",

pages = "116--131",

booktitle = "Proceedings of MT Summit XVI, vol.1: Research Track",

note = "Machine Translation Summit XVI 2017, MT XVI 2017 ; Conference date: 18-09-2017 Through 22-09-2017",

url = "http://www.eamt.org/news/news_cfp_mts2017.php",

}

Castilho, S, Moorkens, J, Gaspari, F, Sennrich, R, Sosoni, V, Georgakopoulou, Y, Lohar, P, Way, A, Miceli Barone, A & Gialama, M 2017, A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators. in Proceedings of MT Summit XVI, vol.1: Research Track. Nagoya, Japan, pp. 116-131, Machine Translation Summit XVI 2017, Nagoya, Japan, 18/09/17. <http://aamt.info/app-def/S-102/mtsummit/2017/wp-content/uploads/sites/2/2017/09/MTSummitXVI_ResearchTrack.pdf>

TY - GEN

T1 - A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators

AU - Castilho, Sheila

AU - Moorkens, Joss

AU - Gaspari, Federico

AU - Sennrich, Rico

AU - Sosoni, Vilelmini

AU - Georgakopoulou, Yota

AU - Lohar, Pintu

AU - Way, Andy

AU - Miceli Barone, Antonio

AU - Gialama, Maria

PY - 2017/9/11

Y1 - 2017/9/11

N2 - This paper reports on a comparative evaluation of phrase-based statistical machine translation (PBSMT) and neural machine translation (NMT) for four language pairs, using the PET interface to compare educational domain output from both systems using a variety of metrics, including automatic evaluation as well as human rankings of adequacy and fluency, error-type markup, and post-editing (technical and temporal) effort, performed by professional translators.Our results show a preference for NMT in side-by-side ranking for all language pairs, texts, and segment lengths. In addition, perceived fluency is improved and annotated errors are fewer in the NMT output. Results are mixed for perceived adequacy and for errors of omission, addition, and mistranslation. Despite far fewer segments requiring post-editing, document-level post-editing performance was not found to have significantly improved in NMT compared toPBSMT. This evaluation was conducted as part of the TraMOOC project, which aims to create a replicable semi-automated methodology for high-quality machine translation of educational data.

AB - This paper reports on a comparative evaluation of phrase-based statistical machine translation (PBSMT) and neural machine translation (NMT) for four language pairs, using the PET interface to compare educational domain output from both systems using a variety of metrics, including automatic evaluation as well as human rankings of adequacy and fluency, error-type markup, and post-editing (technical and temporal) effort, performed by professional translators.Our results show a preference for NMT in side-by-side ranking for all language pairs, texts, and segment lengths. In addition, perceived fluency is improved and annotated errors are fewer in the NMT output. Results are mixed for perceived adequacy and for errors of omission, addition, and mistranslation. Despite far fewer segments requiring post-editing, document-level post-editing performance was not found to have significantly improved in NMT compared toPBSMT. This evaluation was conducted as part of the TraMOOC project, which aims to create a replicable semi-automated methodology for high-quality machine translation of educational data.

M3 - Conference contribution

SP - 116

EP - 131

BT - Proceedings of MT Summit XVI, vol.1: Research Track

CY - Nagoya, Japan

T2 - Machine Translation Summit XVI 2017

Y2 - 18 September 2017 through 22 September 2017

ER -

A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators

Abstract

Conference

Access to Document

Projects

Translation for Massive Open Online Courses- TraMooc

Cite this