Regularization of context-dependent deep neural networks with context-independent multi-task training

Peter Bell; Steve Renals

doi:10.1109/ICASSP.2015.7178780

Regularization of context-dependent deep neural networks with context-independent multi-task training

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The use of context-dependent targets has become standard in hybrid DNN systems for automatic speech recognition. However, we argue that despite the use of state-tying, optimising to context-dependent targets can lead to over-fitting, and that discriminating between arbitrary tied context-dependent targets may not be optimal. We propose a multitask learning method where the network jointly predicts context-dependent and monophone targets. We evaluate the method on a large-vocabulary lecture recognition task and show that it yields relative improvements of 3-10% over baseline systems.

Original language	English
Title of host publication	Proc IEEE International Conference on Acoustics, Speech and Signal Processing
Place of Publication	Brisbane, QLD, Australia
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	4290-4294
Number of pages	5
ISBN (Electronic)	978-1-4673-6997-8
DOIs	https://doi.org/10.1109/ICASSP.2015.7178780
Publication status	Published - 6 Aug 2015
Event	40th IEEE International Conference on Acoustics, Speech and Signal Processing - Brisbane Convention & Exhibition Centre, Brisbane, Australia Duration: 19 Apr 2015 → 24 Apr 2015

Conference

Conference	40th IEEE International Conference on Acoustics, Speech and Signal Processing
Abbreviated title	ICASSP 2015
Country/Territory	Australia
City	Brisbane
Period	19/04/15 → 24/04/15

Access to Document

10.1109/ICASSP.2015.7178780

Bell Renals 2015 Regularization of context-dependent deep neutral networks with context-independent multi-task traingAccepted author manuscript, 145 KB

https://ieeexplore.ieee.org/document/7178780

Natural Speech Technology
Renals, S. & King, S.
EPSRC
1/05/11 → 31/07/16
Project: Research

Peter Bell
- peter.belled.acuk
- School of Informatics - Reader in Speech Technology
- Institute of Language, Cognition and Computation
- Centre for Speech Technology Research
- Language, Interaction and Robotics
Person: Academic: Research Active

Cite this

@inproceedings{1cbf954fc3134fb9b1d1ca18ce682461,

title = "Regularization of context-dependent deep neural networks with context-independent multi-task training",

abstract = "The use of context-dependent targets has become standard in hybrid DNN systems for automatic speech recognition. However, we argue that despite the use of state-tying, optimising to context-dependent targets can lead to over-fitting, and that discriminating between arbitrary tied context-dependent targets may not be optimal. We propose a multitask learning method where the network jointly predicts context-dependent and monophone targets. We evaluate the method on a large-vocabulary lecture recognition task and show that it yields relative improvements of 3-10% over baseline systems.",

author = "Peter Bell and Steve Renals",

year = "2015",

month = aug,

day = "6",

doi = "10.1109/ICASSP.2015.7178780",

language = "English",

pages = "4290--4294",

booktitle = "Proc IEEE International Conference on Acoustics, Speech and Signal Processing",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

address = "United States",

note = "40th IEEE International Conference on Acoustics, Speech and Signal Processing , ICASSP 2015 ; Conference date: 19-04-2015 Through 24-04-2015",

}

Bell, P & Renals, S 2015, Regularization of context-dependent deep neural networks with context-independent multi-task training. in Proc IEEE International Conference on Acoustics, Speech and Signal Processing. Institute of Electrical and Electronics Engineers (IEEE), Brisbane, QLD, Australia, pp. 4290-4294, 40th IEEE International Conference on Acoustics, Speech and Signal Processing , Brisbane, Australia, 19/04/15. https://doi.org/10.1109/ICASSP.2015.7178780

Regularization of context-dependent deep neural networks with context-independent multi-task training. / Bell, Peter ; Renals, Steve.

Proc IEEE International Conference on Acoustics, Speech and Signal Processing. Brisbane, QLD, Australia : Institute of Electrical and Electronics Engineers (IEEE), 2015. p. 4290-4294.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Regularization of context-dependent deep neural networks with context-independent multi-task training

AU - Bell, Peter

AU - Renals, Steve

PY - 2015/8/6

Y1 - 2015/8/6

N2 - The use of context-dependent targets has become standard in hybrid DNN systems for automatic speech recognition. However, we argue that despite the use of state-tying, optimising to context-dependent targets can lead to over-fitting, and that discriminating between arbitrary tied context-dependent targets may not be optimal. We propose a multitask learning method where the network jointly predicts context-dependent and monophone targets. We evaluate the method on a large-vocabulary lecture recognition task and show that it yields relative improvements of 3-10% over baseline systems.

AB - The use of context-dependent targets has become standard in hybrid DNN systems for automatic speech recognition. However, we argue that despite the use of state-tying, optimising to context-dependent targets can lead to over-fitting, and that discriminating between arbitrary tied context-dependent targets may not be optimal. We propose a multitask learning method where the network jointly predicts context-dependent and monophone targets. We evaluate the method on a large-vocabulary lecture recognition task and show that it yields relative improvements of 3-10% over baseline systems.

U2 - 10.1109/ICASSP.2015.7178780

DO - 10.1109/ICASSP.2015.7178780

M3 - Conference contribution

SP - 4290

EP - 4294

BT - Proc IEEE International Conference on Acoustics, Speech and Signal Processing

PB - Institute of Electrical and Electronics Engineers (IEEE)

CY - Brisbane, QLD, Australia

T2 - 40th IEEE International Conference on Acoustics, Speech and Signal Processing

Y2 - 19 April 2015 through 24 April 2015

ER -

Regularization of context-dependent deep neural networks with context-independent multi-task training

Abstract

Conference

Access to Document

Fingerprint

Projects

Natural Speech Technology

Profiles

Peter Bell

Cite this