SPEECH COLLAGE: CODE-SWITCHED AUDIO GENERATION BY COLLAGING MONOLINGUAL CORPORA

Amir Hussein; Dorsa Zeinali; Ondrej Klejch; Matthew Wiesner; Brian Yan; Shammur  Chowdhury; Ahmed Ali; Shinji Watanabe; Sanjeev  Khudanpur

doi:10.1109/ICASSP48485.2024.10446857

SPEECH COLLAGE: CODE-SWITCHED AUDIO GENERATION BY COLLAGING MONOLINGUAL CORPORA

Amir Hussein, Dorsa Zeinali, Ondrej Klejch, Matthew Wiesner, Brian Yan, Shammur Chowdhury, Ahmed Ali, Shinji Watanabe, Sanjeev Khudanpur

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Designing effective automatic speech recognition (ASR) systems for Code-Switching (CS) often depends on the availability of the transcribed CS resources. To address data scarcity, this paper introduces Speech Collage, a method that synthesizes CS data from monolingual corpora by splicing audio segments. We further improve the smoothness quality of audio generation using an overlap-add approach. We investigate the impact of generated data on speech recognition in two scenarios: using in-domain CS text and a zeroshot approach with synthesized CS text. Empirical results highlight up to 34.4% and 16.2% relative reductions in Mixed-Error Rate and Word-Error Rate for in-domain and zero-shot scenarios, respectively. Lastly, we demonstrate that CS augmentation bolsters the model’s code-switching inclination and reduces its monolingual bias.

Original language	English
Title of host publication	ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Publisher	Institute of Electrical and Electronics Engineers
Pages	12006-12010
Number of pages	5
ISBN (Electronic)	979-8-3503-4485-1
DOIs	https://doi.org/10.1109/ICASSP48485.2024.10446857
Publication status	Published - 18 Mar 2024
Event	2024 IEEE International Conference on Acoustics, Speech and Signal Processing - Seoul, Korea, Republic of Duration: 14 Apr 2024 → 19 Apr 2024 https://2024.ieeeicassp.org/

Publication series

Name	International Conference on Acoustics, Speech, and Signal Processing proceedings
Publisher	IEEE
ISSN (Print)	1520-6149
ISSN (Electronic)	2379-190X

Conference

Conference	2024 IEEE International Conference on Acoustics, Speech and Signal Processing
Abbreviated title	ICASSP 2024
Country/Territory	Korea, Republic of
City	Seoul
Period	14/04/24 → 19/04/24
Internet address	https://2024.ieeeicassp.org/

Keywords / Materials (for Non-textual outputs)

Code-switching
ASR
data augmentation
end-to-end
zero-shot learning

Access to Document

10.1109/ICASSP48485.2024.10446857Licence: All Rights Reserved

Speech Collage_HUSSEIN_DOA13122023_AFV_CC BYAccepted author manuscript, 518 KBLicence: Creative Commons: Attribution (CC-BY)

https://ieeexplore.ieee.org/document/10446857Licence: All Rights Reserved

Unmute : Opening Spoken Language Interaction to the Currently Unheard
Bell, P., Goldwater, S. & Renals, S.
EPSRC
1/12/20 → 30/11/23
Project: Research

Cite this

Hussein, A., Zeinali, D., Klejch, O., Wiesner, M., Yan, B., Chowdhury, S., Ali, A., Watanabe, S., & Khudanpur, S. (2024). SPEECH COLLAGE: CODE-SWITCHED AUDIO GENERATION BY COLLAGING MONOLINGUAL CORPORA. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 12006-12010). (International Conference on Acoustics, Speech, and Signal Processing proceedings). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICASSP48485.2024.10446857

Hussein, Amir ; Zeinali, Dorsa ; Klejch, Ondrej et al. / SPEECH COLLAGE: CODE-SWITCHED AUDIO GENERATION BY COLLAGING MONOLINGUAL CORPORA. ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Institute of Electrical and Electronics Engineers, 2024. pp. 12006-12010 (International Conference on Acoustics, Speech, and Signal Processing proceedings).

@inproceedings{e41429f4450045bc989ae42f49c8c6cd,

title = "SPEECH COLLAGE: CODE-SWITCHED AUDIO GENERATION BY COLLAGING MONOLINGUAL CORPORA",

abstract = "Designing effective automatic speech recognition (ASR) systems for Code-Switching (CS) often depends on the availability of the transcribed CS resources. To address data scarcity, this paper introduces Speech Collage, a method that synthesizes CS data from monolingual corpora by splicing audio segments. We further improve the smoothness quality of audio generation using an overlap-add approach. We investigate the impact of generated data on speech recognition in two scenarios: using in-domain CS text and a zeroshot approach with synthesized CS text. Empirical results highlight up to 34.4% and 16.2% relative reductions in Mixed-Error Rate and Word-Error Rate for in-domain and zero-shot scenarios, respectively. Lastly, we demonstrate that CS augmentation bolsters the model{\textquoteright}s code-switching inclination and reduces its monolingual bias.",

keywords = "Code-switching, ASR, data augmentation, end-to-end, zero-shot learning",

author = "Amir Hussein and Dorsa Zeinali and Ondrej Klejch and Matthew Wiesner and Brian Yan and Shammur Chowdhury and Ahmed Ali and Shinji Watanabe and Sanjeev Khudanpur",

year = "2024",

month = mar,

day = "18",

doi = "10.1109/ICASSP48485.2024.10446857",

language = "English",

series = "International Conference on Acoustics, Speech, and Signal Processing proceedings",

publisher = "Institute of Electrical and Electronics Engineers",

pages = "12006--12010",

booktitle = "ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)",

note = "2024 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024 ; Conference date: 14-04-2024 Through 19-04-2024",

url = "https://2024.ieeeicassp.org/",

}

Hussein, A, Zeinali, D, Klejch, O, Wiesner, M, Yan, B, Chowdhury, S, Ali, A, Watanabe, S & Khudanpur, S 2024, SPEECH COLLAGE: CODE-SWITCHED AUDIO GENERATION BY COLLAGING MONOLINGUAL CORPORA. in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). International Conference on Acoustics, Speech, and Signal Processing proceedings, Institute of Electrical and Electronics Engineers, pp. 12006-12010, 2024 IEEE International Conference on Acoustics, Speech and Signal Processing, Seoul, Korea, Republic of, 14/04/24. https://doi.org/10.1109/ICASSP48485.2024.10446857

SPEECH COLLAGE: CODE-SWITCHED AUDIO GENERATION BY COLLAGING MONOLINGUAL CORPORA. / Hussein, Amir; Zeinali, Dorsa; Klejch, Ondrej et al.
ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Institute of Electrical and Electronics Engineers, 2024. p. 12006-12010 (International Conference on Acoustics, Speech, and Signal Processing proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - SPEECH COLLAGE: CODE-SWITCHED AUDIO GENERATION BY COLLAGING MONOLINGUAL CORPORA

AU - Hussein, Amir

AU - Zeinali, Dorsa

AU - Klejch, Ondrej

AU - Wiesner, Matthew

AU - Yan, Brian

AU - Chowdhury, Shammur

AU - Ali, Ahmed

AU - Watanabe, Shinji

AU - Khudanpur, Sanjeev

PY - 2024/3/18

Y1 - 2024/3/18

N2 - Designing effective automatic speech recognition (ASR) systems for Code-Switching (CS) often depends on the availability of the transcribed CS resources. To address data scarcity, this paper introduces Speech Collage, a method that synthesizes CS data from monolingual corpora by splicing audio segments. We further improve the smoothness quality of audio generation using an overlap-add approach. We investigate the impact of generated data on speech recognition in two scenarios: using in-domain CS text and a zeroshot approach with synthesized CS text. Empirical results highlight up to 34.4% and 16.2% relative reductions in Mixed-Error Rate and Word-Error Rate for in-domain and zero-shot scenarios, respectively. Lastly, we demonstrate that CS augmentation bolsters the model’s code-switching inclination and reduces its monolingual bias.

AB - Designing effective automatic speech recognition (ASR) systems for Code-Switching (CS) often depends on the availability of the transcribed CS resources. To address data scarcity, this paper introduces Speech Collage, a method that synthesizes CS data from monolingual corpora by splicing audio segments. We further improve the smoothness quality of audio generation using an overlap-add approach. We investigate the impact of generated data on speech recognition in two scenarios: using in-domain CS text and a zeroshot approach with synthesized CS text. Empirical results highlight up to 34.4% and 16.2% relative reductions in Mixed-Error Rate and Word-Error Rate for in-domain and zero-shot scenarios, respectively. Lastly, we demonstrate that CS augmentation bolsters the model’s code-switching inclination and reduces its monolingual bias.

KW - Code-switching

KW - ASR

KW - data augmentation

KW - end-to-end

KW - zero-shot learning

U2 - 10.1109/ICASSP48485.2024.10446857

DO - 10.1109/ICASSP48485.2024.10446857

M3 - Conference contribution

T3 - International Conference on Acoustics, Speech, and Signal Processing proceedings

SP - 12006

EP - 12010

BT - ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

PB - Institute of Electrical and Electronics Engineers

T2 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing

Y2 - 14 April 2024 through 19 April 2024

ER -

Hussein A, Zeinali D, Klejch O, Wiesner M, Yan B, Chowdhury S et al. SPEECH COLLAGE: CODE-SWITCHED AUDIO GENERATION BY COLLAGING MONOLINGUAL CORPORA. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Institute of Electrical and Electronics Engineers. 2024. p. 12006-12010. (International Conference on Acoustics, Speech, and Signal Processing proceedings). doi: 10.1109/ICASSP48485.2024.10446857

SPEECH COLLAGE: CODE-SWITCHED AUDIO GENERATION BY COLLAGING MONOLINGUAL CORPORA

Abstract / Description of output

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Projects

Unmute : Opening Spoken Language Interaction to the Currently Unheard

Cite this