What does BERT learn about prosody?

Sofoklis Kakouros, Johannah O'Mahony

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Language models have become nearly ubiquitous innatural language processing applications achieving state-of-the-art results in many tasks including prosody. As the model design does not define predetermined linguistic targets during training but rather aims at learning generalized representations of the language, analyzing and interpreting there presentations that models implicitly capture is important in bridging the gap between interpretability and model performance. Several studies have explored the linguistic information that models capture providing some insights on their representational capacity. However, the current studies have not explored whether prosody is part of the structural information of the language that models learn. In this work, we perform a series of experiments on BERT probing the representations captured at different layers. Our results show that information about prosodic prominence spans across many layers but is mostly focused in middle layers suggesting that BERT relies mostly on syntactic and semantic information.
Original languageEnglish
Title of host publicationProceedings of the 20th International Congress of Phonetic Sciences
EditorsRadek Skarnitzl, Jan Volín
Place of PublicationPrague
PublisherGuarant International
Pages1454-1458
ISBN (Electronic)9788090811423
Publication statusPublished - 10 Aug 2023
Event20th International Conference of Phonetic Sciences (ICPhS) - Prague Congress Centre, Prague, Czech Republic
Duration: 7 Aug 202311 Aug 2023
https://www.icphs2023.org/

Conference

Conference20th International Conference of Phonetic Sciences (ICPhS)
Abbreviated titleICPhS 2023
Country/TerritoryCzech Republic
CityPrague
Period7/08/2311/08/23
Internet address

Keywords / Materials (for Non-textual outputs)

  • language model
  • BERT
  • prosody
  • prominence
  • part-of-speech

Fingerprint

Dive into the research topics of 'What does BERT learn about prosody?'. Together they form a unique fingerprint.

Cite this