Intonation control for neural text-to-speech synthesis with polynomial models of F0

Niamh Corkey; Johannah O'Mahony; Simon King

Intonation control for neural text-to-speech synthesis with polynomial models of F0

Niamh Corkey, Johannah O'Mahony, Simon King

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

We present a novel, user-friendly approach for controlling patterns of intonation (a fundamental aspect of prosody) within a neural TTS system. This involves concisely representing F0 contours with the coefficients of their Legendre polynomial series expansion, and implementing a model (based on FastPitch) which is conditioned on these sets of coefficients during training. At inference time the model will explicitly predict a coefficient set, or a user (eg. human-in-the-loop) can provide a target coefficient set which audibly alters the intonation of the output speech, based on just a few values. This is particularly effective for intonation transfer: where these coefficient targets are extracted from a ground truth recording, making the synthesised utterance more closely reflect the intonation of the real speaker.

Original language	English
Title of host publication	Proceedings of the Annual Conference of the International Speech Communication Association
Subtitle of host publication	Interspeech 2023
Editors	Naomi Harte, Julie Carson-Berndsen, Gareth Jones
Place of Publication	Dublin
Publisher	ISCA
Pages	2014-2015
Number of pages	2
Publication status	Published - Sept 2023
Event	Interspeech 2023 - Dublin, Ireland Duration: 20 Aug 2023 → 24 Aug 2023 Conference number: 24 https://www.interspeech2023.org/

Publication series

Name	Interspeech - Annual Conference of the International Speech Communication Association
Publisher	ISCA
ISSN (Electronic)	2308-457X

Conference

Conference	Interspeech 2023
Country/Territory	Ireland
City	Dublin
Period	20/08/23 → 24/08/23
Internet address	https://www.interspeech2023.org/

Keywords / Materials (for Non-textual outputs)

text-to-speech
speech synthesis
intonation modelling
prosody control
prosody transfer

Access to Document

CorkeyEtAl2023IntonationControlForNeuralAccepted author manuscript, 196 KBLicence: Creative Commons: Attribution (CC-BY)

https://www.isca-speech.org/archive/interspeech_2023/corkey23_interspeech.html

Cite this

Corkey, N., O'Mahony, J., & King, S. (2023). Intonation control for neural text-to-speech synthesis with polynomial models of F0. In N. Harte, J. Carson-Berndsen, & G. Jones (Eds.), Proceedings of the Annual Conference of the International Speech Communication Association: Interspeech 2023 (pp. 2014-2015). (Interspeech - Annual Conference of the International Speech Communication Association). ISCA. https://www.isca-speech.org/archive/interspeech_2023/corkey23_interspeech.html

Corkey, Niamh ; O'Mahony, Johannah ; King, Simon. / Intonation control for neural text-to-speech synthesis with polynomial models of F0. Proceedings of the Annual Conference of the International Speech Communication Association: Interspeech 2023. editor / Naomi Harte ; Julie Carson-Berndsen ; Gareth Jones. Dublin : ISCA, 2023. pp. 2014-2015 (Interspeech - Annual Conference of the International Speech Communication Association).

@inproceedings{f3f8d0443192481aa0686bab72132ff4,

title = "Intonation control for neural text-to-speech synthesis with polynomial models of F0",

abstract = "We present a novel, user-friendly approach for controlling patterns of intonation (a fundamental aspect of prosody) within a neural TTS system. This involves concisely representing F0 contours with the coefficients of their Legendre polynomial series expansion, and implementing a model (based on FastPitch) which is conditioned on these sets of coefficients during training. At inference time the model will explicitly predict a coefficient set, or a user (eg. human-in-the-loop) can provide a target coefficient set which audibly alters the intonation of the output speech, based on just a few values. This is particularly effective for intonation transfer: where these coefficient targets are extracted from a ground truth recording, making the synthesised utterance more closely reflect the intonation of the real speaker.",

keywords = "text-to-speech, speech synthesis, intonation modelling, prosody control, prosody transfer",

author = "Niamh Corkey and Johannah O'Mahony and Simon King",

note = "Acknowledgements: This project has received funding from the European Union{\textquoteright}s Horizon 2020 research and innovation programme under the Marie Sk{\l}odowska-Curie grant agreement No 859588.; Interspeech 2023 ; Conference date: 20-08-2023 Through 24-08-2023",

year = "2023",

month = sep,

language = "English",

series = "Interspeech - Annual Conference of the International Speech Communication Association",

publisher = "ISCA",

pages = "2014--2015",

editor = "Naomi Harte and Julie Carson-Berndsen and Gareth Jones",

booktitle = "Proceedings of the Annual Conference of the International Speech Communication Association",

url = "https://www.interspeech2023.org/",

}

Corkey, N, O'Mahony, J & King, S 2023, Intonation control for neural text-to-speech synthesis with polynomial models of F0. in N Harte, J Carson-Berndsen & G Jones (eds), Proceedings of the Annual Conference of the International Speech Communication Association: Interspeech 2023. Interspeech - Annual Conference of the International Speech Communication Association, ISCA, Dublin, pp. 2014-2015, Interspeech 2023, Dublin, Ireland, 20/08/23. <https://www.isca-speech.org/archive/interspeech_2023/corkey23_interspeech.html>

Intonation control for neural text-to-speech synthesis with polynomial models of F0. / Corkey, Niamh; O'Mahony, Johannah; King, Simon.
Proceedings of the Annual Conference of the International Speech Communication Association: Interspeech 2023. ed. / Naomi Harte; Julie Carson-Berndsen; Gareth Jones. Dublin: ISCA, 2023. p. 2014-2015 (Interspeech - Annual Conference of the International Speech Communication Association).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Intonation control for neural text-to-speech synthesis with polynomial models of F0

AU - Corkey, Niamh

AU - O'Mahony, Johannah

AU - King, Simon

N1 - Conference code: 24

PY - 2023/9

Y1 - 2023/9

N2 - We present a novel, user-friendly approach for controlling patterns of intonation (a fundamental aspect of prosody) within a neural TTS system. This involves concisely representing F0 contours with the coefficients of their Legendre polynomial series expansion, and implementing a model (based on FastPitch) which is conditioned on these sets of coefficients during training. At inference time the model will explicitly predict a coefficient set, or a user (eg. human-in-the-loop) can provide a target coefficient set which audibly alters the intonation of the output speech, based on just a few values. This is particularly effective for intonation transfer: where these coefficient targets are extracted from a ground truth recording, making the synthesised utterance more closely reflect the intonation of the real speaker.

AB - We present a novel, user-friendly approach for controlling patterns of intonation (a fundamental aspect of prosody) within a neural TTS system. This involves concisely representing F0 contours with the coefficients of their Legendre polynomial series expansion, and implementing a model (based on FastPitch) which is conditioned on these sets of coefficients during training. At inference time the model will explicitly predict a coefficient set, or a user (eg. human-in-the-loop) can provide a target coefficient set which audibly alters the intonation of the output speech, based on just a few values. This is particularly effective for intonation transfer: where these coefficient targets are extracted from a ground truth recording, making the synthesised utterance more closely reflect the intonation of the real speaker.

KW - text-to-speech

KW - speech synthesis

KW - intonation modelling

KW - prosody control

KW - prosody transfer

UR - https://github.com/evdv/FastPitches

M3 - Conference contribution

T3 - Interspeech - Annual Conference of the International Speech Communication Association

SP - 2014

EP - 2015

BT - Proceedings of the Annual Conference of the International Speech Communication Association

A2 - Harte, Naomi

A2 - Carson-Berndsen, Julie

A2 - Jones, Gareth

PB - ISCA

CY - Dublin

T2 - Interspeech 2023

Y2 - 20 August 2023 through 24 August 2023

ER -

Corkey N, O'Mahony J, King S. Intonation control for neural text-to-speech synthesis with polynomial models of F0. In Harte N, Carson-Berndsen J, Jones G, editors, Proceedings of the Annual Conference of the International Speech Communication Association: Interspeech 2023. Dublin: ISCA. 2023. p. 2014-2015. (Interspeech - Annual Conference of the International Speech Communication Association). Epub 2023 Aug 24.

Intonation control for neural text-to-speech synthesis with polynomial models of F0

Abstract / Description of output

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Other files and links

Fingerprint

COBRA: COnversational BRAins

Cite this

Intonation control for neural text-to-speech synthesis with polynomial models of F0

Abstract / Description of output

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Other files and links

Fingerprint

Projects

COBRA: COnversational BRAins

Cite this