A Study in Improving BLEU Reference Coverage with Diverse Automatic Paraphrasing

Rachel Bawden, Biao Zhang, Lisa Yankovskaya, Andre Tättar, Matt Post

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We investigate a long-perceived shortcoming in the typical use of BLEU: its reliance on a single reference. Using modern neural paraphrasing techniques, we study whether automatically generating additional diverse references can provide better coverage of the space of valid translations and thereby improve its correlation with human judgments. Our experiments on the into-English language directions of the WMT19 metrics task (at both the system and sentence level) show that using paraphrased references does generally improve BLEU, and when it does, the more diverse the better. However, we also show that better results could be achieved if those paraphrases were to specifically target the parts of the space most relevant to the MT outputs being evaluated. Moreover, the gains remain slight even when using human paraphrases elicited to maximize diversity, suggesting inherent limitations to BLEU's capacity to correctly exploit multiple references. Surprisingly , we also find that adequacy appears to be less important, as shown by the high results of a strong sampling approach, which even beats human paraphrases when used with sentence-level BLEU.
Original languageEnglish
Title of host publicationFindings of the Association for Computational Linguistics: EMNLP 2020
PublisherAssociation for Computational Linguistics (ACL)
Pages918-932
Number of pages15
ISBN (Electronic)978-1-952148-90-3
Publication statusPublished - 16 Nov 2020
EventThe 2020 Conference on Empirical Methods in Natural Language Processing - Virtual conference
Duration: 16 Nov 202020 Nov 2020
https://2020.emnlp.org/

Conference

ConferenceThe 2020 Conference on Empirical Methods in Natural Language Processing
Abbreviated titleEMNLP 2020
CityVirtual conference
Period16/11/2020/11/20
Internet address

Keywords

  • machine translation
  • evaluation
  • metrics
  • paraphrasing
  • BLEU

Fingerprint Dive into the research topics of 'A Study in Improving BLEU Reference Coverage with Diverse Automatic Paraphrasing'. Together they form a unique fingerprint.

Cite this