Iterative translation refinement with large language models

Pinzhen Chen; Zhicheng Guo; Barry Haddow; Kenneth Heafield

Iterative translation refinement with large language models

Pinzhen Chen, Zhicheng Guo, Barry Haddow, Kenneth Heafield

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

We propose iteratively prompting a large language model to self-correct a translation, with inspiration from their strong language capability as well as a human-like translation approach. Interestingly, multi-turn querying reduces the output's string-based metric scores, but neural metrics suggest comparable or improved quality after two or more iterations. Human evaluations indicate better fluency and naturalness compared to initial translations and even human references, all while maintaining quality. Ablation studies underscore the importance of anchoring the refinement to the source and a reasonable seed translation for quality considerations. We also discuss the challenges in evaluation and relation to human performance and translationese.

Original language	English
Title of host publication	Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1)
Editors	Carolina Scarton, Charlotte Prescott, Chris Bayliss, Chris Oakley, Joanna Wright, Stuart Wrigley, Xingyi Song, Edward Gow-Smith, Rachel Bawden, Victor M Sánchez-Cartagena, Patrick Cadwell, Ekaterina Lapshinova-Koltunski, Vera Cabarrão, Konstantinos Chatzitheodorou, Mary Nurminen, Diptesh Kanojia, Helena Moniz
Publisher	European Association for Machine Translation (EAMT)
Pages	181-190
Number of pages	10
Volume	1
ISBN (Electronic)	9781068690709
Publication status	Published - 27 Jun 2024
Event	The 25th Annual Conference of The European Association for Machine Translation - University of Sheffield, Sheffield, United Kingdom Duration: 24 Jun 2024 → 27 Jun 2024 Conference number: 25 https://eamt2024.sheffield.ac.uk/

Conference

Conference	The 25th Annual Conference of The European Association for Machine Translation
Abbreviated title	EAMT 2024
Country/Territory	United Kingdom
City	Sheffield
Period	24/06/24 → 27/06/24
Internet address	https://eamt2024.sheffield.ac.uk/

Access to Document

ChenEtalEAMT2024IterativeTranslationRefinementFinal published version, 532 KBLicence: Creative Commons: Attribution No Derivatives (CC-BY-ND)

https://aclanthology.org/2024.eamt-1.17Licence: Creative Commons: Attribution No Derivatives (CC-BY-ND)

Cite this

Chen, P., Guo, Z., Haddow, B., & Heafield, K. (2024). Iterative translation refinement with large language models. In C. Scarton, C. Prescott, C. Bayliss, C. Oakley, J. Wright, S. Wrigley, X. Song, E. Gow-Smith, R. Bawden, V. M. Sánchez-Cartagena, P. Cadwell, E. Lapshinova-Koltunski, V. Cabarrão, K. Chatzitheodorou, M. Nurminen, D. Kanojia, & H. Moniz (Eds.), Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1) (Vol. 1, pp. 181-190). European Association for Machine Translation (EAMT). https://aclanthology.org/2024.eamt-1.17

Chen, Pinzhen ; Guo, Zhicheng ; Haddow, Barry et al. / Iterative translation refinement with large language models. Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1). editor / Carolina Scarton ; Charlotte Prescott ; Chris Bayliss ; Chris Oakley ; Joanna Wright ; Stuart Wrigley ; Xingyi Song ; Edward Gow-Smith ; Rachel Bawden ; Victor M Sánchez-Cartagena ; Patrick Cadwell ; Ekaterina Lapshinova-Koltunski ; Vera Cabarrão ; Konstantinos Chatzitheodorou ; Mary Nurminen ; Diptesh Kanojia ; Helena Moniz. Vol. 1 European Association for Machine Translation (EAMT), 2024. pp. 181-190

@inproceedings{8efbdf7322954d2091bc584db1a65e28,

title = "Iterative translation refinement with large language models",

abstract = "We propose iteratively prompting a large language model to self-correct a translation, with inspiration from their strong language capability as well as a human-like translation approach. Interestingly, multi-turn querying reduces the output's string-based metric scores, but neural metrics suggest comparable or improved quality after two or more iterations. Human evaluations indicate better fluency and naturalness compared to initial translations and even human references, all while maintaining quality. Ablation studies underscore the importance of anchoring the refinement to the source and a reasonable seed translation for quality considerations. We also discuss the challenges in evaluation and relation to human performance and translationese.",

author = "Pinzhen Chen and Zhicheng Guo and Barry Haddow and Kenneth Heafield",

year = "2024",

month = jun,

day = "27",

language = "English",

volume = "1",

pages = "181--190",

editor = "Carolina Scarton and Charlotte Prescott and Chris Bayliss and Chris Oakley and Joanna Wright and Stuart Wrigley and Xingyi Song and Edward Gow-Smith and Rachel Bawden and S{\'a}nchez-Cartagena, {Victor M} and Patrick Cadwell and Ekaterina Lapshinova-Koltunski and Vera Cabarr{\~a}o and Konstantinos Chatzitheodorou and Mary Nurminen and Diptesh Kanojia and Helena Moniz",

booktitle = "Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1)",

publisher = "European Association for Machine Translation (EAMT)",

note = "The 25th Annual Conference of The European Association for Machine Translation, EAMT 2024 ; Conference date: 24-06-2024 Through 27-06-2024",

url = "https://eamt2024.sheffield.ac.uk/",

}

Chen, P, Guo, Z, Haddow, B & Heafield, K 2024, Iterative translation refinement with large language models. in C Scarton, C Prescott, C Bayliss, C Oakley, J Wright, S Wrigley, X Song, E Gow-Smith, R Bawden, VM Sánchez-Cartagena, P Cadwell, E Lapshinova-Koltunski, V Cabarrão, K Chatzitheodorou, M Nurminen, D Kanojia & H Moniz (eds), Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1). vol. 1, European Association for Machine Translation (EAMT), pp. 181-190, The 25th Annual Conference of The European Association for Machine Translation, Sheffield, United Kingdom, 24/06/24. <https://aclanthology.org/2024.eamt-1.17>

Iterative translation refinement with large language models. / Chen, Pinzhen; Guo, Zhicheng; Haddow, Barry et al.
Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1). ed. / Carolina Scarton; Charlotte Prescott; Chris Bayliss; Chris Oakley; Joanna Wright; Stuart Wrigley; Xingyi Song; Edward Gow-Smith; Rachel Bawden; Victor M Sánchez-Cartagena; Patrick Cadwell; Ekaterina Lapshinova-Koltunski; Vera Cabarrão; Konstantinos Chatzitheodorou; Mary Nurminen; Diptesh Kanojia; Helena Moniz. Vol. 1 European Association for Machine Translation (EAMT), 2024. p. 181-190.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Iterative translation refinement with large language models

AU - Chen, Pinzhen

AU - Guo, Zhicheng

AU - Haddow, Barry

AU - Heafield, Kenneth

N1 - Conference code: 25

PY - 2024/6/27

Y1 - 2024/6/27

N2 - We propose iteratively prompting a large language model to self-correct a translation, with inspiration from their strong language capability as well as a human-like translation approach. Interestingly, multi-turn querying reduces the output's string-based metric scores, but neural metrics suggest comparable or improved quality after two or more iterations. Human evaluations indicate better fluency and naturalness compared to initial translations and even human references, all while maintaining quality. Ablation studies underscore the importance of anchoring the refinement to the source and a reasonable seed translation for quality considerations. We also discuss the challenges in evaluation and relation to human performance and translationese.

AB - We propose iteratively prompting a large language model to self-correct a translation, with inspiration from their strong language capability as well as a human-like translation approach. Interestingly, multi-turn querying reduces the output's string-based metric scores, but neural metrics suggest comparable or improved quality after two or more iterations. Human evaluations indicate better fluency and naturalness compared to initial translations and even human references, all while maintaining quality. Ablation studies underscore the importance of anchoring the refinement to the source and a reasonable seed translation for quality considerations. We also discuss the challenges in evaluation and relation to human performance and translationese.

M3 - Conference contribution

VL - 1

SP - 181

EP - 190

BT - Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1)

A2 - Scarton, Carolina

A2 - Prescott, Charlotte

A2 - Bayliss, Chris

A2 - Oakley, Chris

A2 - Wright, Joanna

A2 - Wrigley, Stuart

A2 - Song, Xingyi

A2 - Gow-Smith, Edward

A2 - Bawden, Rachel

A2 - Sánchez-Cartagena, Victor M

A2 - Cadwell, Patrick

A2 - Lapshinova-Koltunski, Ekaterina

A2 - Cabarrão, Vera

A2 - Chatzitheodorou, Konstantinos

A2 - Nurminen, Mary

A2 - Kanojia, Diptesh

A2 - Moniz, Helena

PB - European Association for Machine Translation (EAMT)

T2 - The 25th Annual Conference of The European Association for Machine Translation

Y2 - 24 June 2024 through 27 June 2024

ER -

Chen P, Guo Z, Haddow B, Heafield K. Iterative translation refinement with large language models. In Scarton C, Prescott C, Bayliss C, Oakley C, Wright J, Wrigley S, Song X, Gow-Smith E, Bawden R, Sánchez-Cartagena VM, Cadwell P, Lapshinova-Koltunski E, Cabarrão V, Chatzitheodorou K, Nurminen M, Kanojia D, Moniz H, editors, Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1). Vol. 1. European Association for Machine Translation (EAMT). 2024. p. 181-190

Iterative translation refinement with large language models

Abstract / Description of output

Conference

Access to Document

Fingerprint

Cite this