Prosody in cascade and direct speech-to-text translation: A case study on Korean wh-phrases

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Speech-to-Text Translation (S2TT) has typically been addressed with cascade systems, where speech recognition systems generate a transcription that is subsequently passed to a translation model. While there has been a growing interest in developing direct speech translation systems to avoid propagating errors and losing non-verbal content, prior work in direct S2TT has struggled to conclusively establish the advantages of integrating the acoustic signal directly into the translation process. This work proposes using contrastive evaluation to quantitatively measure the ability of direct S2TT systems to disambiguate utterances where prosody plays a crucial role. Specifically, we evaluated Korean-English translation systems on a test set containing wh-phrases, for which prosodic features are necessary to produce translations with the correct intent, whether it’s a statement, a yes/no question, a wh-question, and more. Our results clearly demonstrate the value of direct translation systems over cascade translation models, with a notable 12.9% improvement in overall accuracy in ambiguous cases, along with up to a 15.6% increase in F1 scores for one of the major intent categories. To the best of our knowledge, this work stands as the first to provide quantitative evidence that direct S2TT models can effectively leverage prosody. The code for our evaluation is openly accessible and freely available for review and utilisation.
Original languageEnglish
Title of host publicationFindings of the Association for Computational Linguistics
Subtitle of host publicationEACL 2024
EditorsYvette Graham, Matthew Purver
PublisherAssociation for Computational Linguistics
Pages674–683
Number of pages10
ISBN (Electronic)9798891760936
Publication statusPublished - 2 Mar 2024
EventThe 18th Conference of the European Chapter of the Association for Computational Linguistics - St. Julian’s, Malta
Duration: 17 Mar 202422 Mar 2024
Conference number: 18
https://2024.eacl.org/

Conference

ConferenceThe 18th Conference of the European Chapter of the Association for Computational Linguistics
Abbreviated titleEACL 2024
Country/TerritoryMalta
CitySt. Julian’s
Period17/03/2422/03/24
Internet address

Fingerprint

Dive into the research topics of 'Prosody in cascade and direct speech-to-text translation: A case study on Korean wh-phrases'. Together they form a unique fingerprint.

Cite this