Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora

Yuanchao Li; Yumnah Mohamied; Peter Bell; Catherine Lai

doi:10.1109/SLT54892.2023.10023428

Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora

Yuanchao Li^*, Yumnah Mohamied, Peter Bell, Catherine Lai

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Self-supervised speech models have grown fast during the past few years and have proven feasible for use in various downstream tasks. Some recent work has started to look at the characteristics of these models, yet many concerns have not been fully addressed. In this work, we conduct a study on emotional corpora to explore a popular self-supervised model -- wav2vec 2.0. Via a set of quantitative analysis, we mainly demonstrate that: 1) wav2vec 2.0 appears to discard paralinguistic information that is less useful for word recognition purposes; 2) for emotion recognition, representations from the middle layer alone perform as well as those derived from layer averaging, while the final layer results in the worst performance in some cases; 3) current self-supervised models may not be the optimal solution for downstream tasks that make use of non-lexical features. Our work provides novel findings that will aid future research in this area and theoretical basis for the use of existing models.

Original language	English
Title of host publication	Proceedings of the 2022 IEEE Spoken Language Technology Workshop
Publisher	Institute of Electrical and Electronics Engineers
Pages	868-875
Number of pages	8
ISBN (Electronic)	979-8-3503-9690-4, 979-8-3503-9689-8
ISBN (Print)	979-8-3503-9691-1
DOIs	https://doi.org/10.1109/SLT54892.2023.10023428
Publication status	Published - 27 Jan 2023
Event	The IEEE Spoken Language Technology Workshop, 2022 - Doha, Qatar Duration: 9 Jan 2023 → 12 Jan 2023 https://slt2022.org/

Workshop

Workshop	The IEEE Spoken Language Technology Workshop, 2022
Abbreviated title	SLT 2022
Country/Territory	Qatar
City	Doha
Period	9/01/23 → 12/01/23
Internet address	https://slt2022.org/

Keywords / Materials (for Non-textual outputs)

wav2vec 2.0
self-supervised learning
speech emotion
speech recognition
paralinguistics

Access to Document

10.1109/SLT54892.2023.10023428Licence: All Rights Reserved

SLT_2022-finalAccepted author manuscript, 1.49 MBLicence: Creative Commons: Attribution (CC-BY)

https://ieeexplore.ieee.org/document/10023428Licence: All Rights Reserved

Cite this

@inproceedings{7fdf9f7069084f69a6e57b3097e46fc4,

title = "Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora",

abstract = "Self-supervised speech models have grown fast during the past few years and have proven feasible for use in various downstream tasks. Some recent work has started to look at the characteristics of these models, yet many concerns have not been fully addressed. In this work, we conduct a study on emotional corpora to explore a popular self-supervised model -- wav2vec 2.0. Via a set of quantitative analysis, we mainly demonstrate that: 1) wav2vec 2.0 appears to discard paralinguistic information that is less useful for word recognition purposes; 2) for emotion recognition, representations from the middle layer alone perform as well as those derived from layer averaging, while the final layer results in the worst performance in some cases; 3) current self-supervised models may not be the optimal solution for downstream tasks that make use of non-lexical features. Our work provides novel findings that will aid future research in this area and theoretical basis for the use of existing models.",

keywords = "wav2vec 2.0, self-supervised learning, speech emotion, speech recognition, paralinguistics",

author = "Yuanchao Li and Yumnah Mohamied and Peter Bell and Catherine Lai",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; The IEEE Spoken Language Technology Workshop, 2022, SLT 2022 ; Conference date: 09-01-2023 Through 12-01-2023",

year = "2023",

month = jan,

day = "27",

doi = "10.1109/SLT54892.2023.10023428",

language = "English",

isbn = "979-8-3503-9691-1",

pages = "868--875",

booktitle = "Proceedings of the 2022 IEEE Spoken Language Technology Workshop",

publisher = "Institute of Electrical and Electronics Engineers",

url = "https://slt2022.org/",

}

Li, Y, Mohamied, Y, Bell, P & Lai, C 2023, Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora. in Proceedings of the 2022 IEEE Spoken Language Technology Workshop. Institute of Electrical and Electronics Engineers, pp. 868-875, The IEEE Spoken Language Technology Workshop, 2022, Doha, Qatar, 9/01/23. https://doi.org/10.1109/SLT54892.2023.10023428

TY - GEN

T1 - Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora

AU - Li, Yuanchao

AU - Mohamied, Yumnah

AU - Bell, Peter

AU - Lai, Catherine

PY - 2023/1/27

Y1 - 2023/1/27

N2 - Self-supervised speech models have grown fast during the past few years and have proven feasible for use in various downstream tasks. Some recent work has started to look at the characteristics of these models, yet many concerns have not been fully addressed. In this work, we conduct a study on emotional corpora to explore a popular self-supervised model -- wav2vec 2.0. Via a set of quantitative analysis, we mainly demonstrate that: 1) wav2vec 2.0 appears to discard paralinguistic information that is less useful for word recognition purposes; 2) for emotion recognition, representations from the middle layer alone perform as well as those derived from layer averaging, while the final layer results in the worst performance in some cases; 3) current self-supervised models may not be the optimal solution for downstream tasks that make use of non-lexical features. Our work provides novel findings that will aid future research in this area and theoretical basis for the use of existing models.

AB - Self-supervised speech models have grown fast during the past few years and have proven feasible for use in various downstream tasks. Some recent work has started to look at the characteristics of these models, yet many concerns have not been fully addressed. In this work, we conduct a study on emotional corpora to explore a popular self-supervised model -- wav2vec 2.0. Via a set of quantitative analysis, we mainly demonstrate that: 1) wav2vec 2.0 appears to discard paralinguistic information that is less useful for word recognition purposes; 2) for emotion recognition, representations from the middle layer alone perform as well as those derived from layer averaging, while the final layer results in the worst performance in some cases; 3) current self-supervised models may not be the optimal solution for downstream tasks that make use of non-lexical features. Our work provides novel findings that will aid future research in this area and theoretical basis for the use of existing models.

KW - wav2vec 2.0

KW - self-supervised learning

KW - speech emotion

KW - speech recognition

KW - paralinguistics

U2 - 10.1109/SLT54892.2023.10023428

DO - 10.1109/SLT54892.2023.10023428

M3 - Conference contribution

SN - 979-8-3503-9691-1

SP - 868

EP - 875

BT - Proceedings of the 2022 IEEE Spoken Language Technology Workshop

PB - Institute of Electrical and Electronics Engineers

T2 - The IEEE Spoken Language Technology Workshop, 2022

Y2 - 9 January 2023 through 12 January 2023

ER -

Exploration of A Self-Supervised Speech Model: A Study on Emotional Corpora

Abstract / Description of output

Workshop

Keywords / Materials (for Non-textual outputs)

Access to Document

Fingerprint

Cite this