Fusing ASR Outputs in Joint Training for Speech Emotion Recognition

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Alongside acoustic information, linguistic features based on speech transcripts have been proven useful in Speech Emotion Recognition (SER). However, due to the scarcity of emotion labelled data and the difficulty of recognizing emotional speech, it is hard to obtain reliable linguistic features and models in this research area. In this paper, we propose to fuse Automatic Speech Recognition (ASR) outputs into the pipeline for joint training SER. The relationship between ASR and SER is understudied, and it is unclear what and how ASR features benefit SER. By examining various ASR outputs and fusion methods, our experiments show that in joint ASR-SER training, incorporating both ASR hidden and text output using a hierarchical co-attention fusion approach improves the SER performance the most. On the IEMOCAP corpus, our approach achieves 63.4\% weighted accuracy, which is close to the baseline results achieved by combining ground-truth transcripts. In addition, we also present novel word error rate analysis on IEMOCAP and layer-difference analysis of the Wav2vec 2.0 model to better understand the relationship between ASR and SER.
Original languageEnglish
Title of host publicationProceedings of 2022 IEEE International Conference on Acoustics, Speech and Signal Processing
PublisherInstitute of Electrical and Electronics Engineers
Pages7362-7366
Number of pages5
ISBN (Electronic)978-1-6654-0540-9
ISBN (Print)978-1-6654-0541-6
DOIs
Publication statusPublished - 27 Apr 2022
Event2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Online, Singapore
Duration: 7 May 202227 May 2022
Conference number: 47
https://2022.ieeeicassp.org/index.php

Publication series

NameInternational Conference on Acoustics, Speech, and Signal Processing (ICASSP)
PublisherIEEE
ISSN (Print)1520-6149
ISSN (Electronic)2379-190X

Conference

Conference2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Abbreviated titleICASSP 2022
Period7/05/2227/05/22
Internet address

Keywords / Materials (for Non-textual outputs)

  • Speech emotion recognition
  • Automatic speech recognition
  • Multi-task learning
  • Wav2vec 2.0

Fingerprint

Dive into the research topics of 'Fusing ASR Outputs in Joint Training for Speech Emotion Recognition'. Together they form a unique fingerprint.

Cite this