On the Usefulness of Statistical Normalisation of Bottleneck Features for Speech Recognition

Erfan Loweimi; Peter Bell; Steve Renals

doi:10.1109/ICASSP.2019.8683330

On the Usefulness of Statistical Normalisation of Bottleneck Features for Speech Recognition

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

DNNs play a major role in the state-of-the-art ASR systems. They can be used for extracting features and building probabilistic models for acoustic and language modelling. Despite their huge practical success, the level of theoretical understanding has remained shallow. This paper investigates DNNs from a statistical standpoint. In particular, the effect of activation functions on the distribution of the pre-activations and activations is investigated and discussed from both analytic and empirical viewpoints. This study, among others, shows that the pre-activation density in the bottleneck layer can be well fitted with a diagonal GMM with a few Gaussians and how and why the ReLU activation function promotes sparsity. Motivated by the statistical properties of the pre-activations, the usefulness of statistical normalisation of bottleneck features was also investigated. To this end, methods such as mean(-variance) normalisation, Gaussianisation, and histogram equalisation (HEQ) were employed and up to 2% (absolute) WER reduction achieved in the Aurora-4 task.

Original language	English
Title of host publication	ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Place of Publication	Brighton, United Kingdom
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Pages	3862-3866
Number of pages	5
ISBN (Electronic)	978-1-4799-8131-1
ISBN (Print)	978-1-4799-8132-8
DOIs	https://doi.org/10.1109/ICASSP.2019.8683330
Publication status	E-pub ahead of print - 17 Apr 2019
Event	44th International Conference on Acoustics, Speech, and Signal Processing: Signal Processing: Empowering Science and Technology for Humankind - Brighton , United Kingdom Duration: 12 May 2019 → 17 May 2019 Conference number: 44 https://2019.ieeeicassp.org/

Publication series

Name
Publisher	IEEE
ISSN (Print)	1520-6149
ISSN (Electronic)	2379-190X

Conference

Conference	44th International Conference on Acoustics, Speech, and Signal Processing
Abbreviated title	ICASSP 2019
Country/Territory	United Kingdom
City	Brighton
Period	12/05/19 → 17/05/19
Internet address	https://2019.ieeeicassp.org/

Keywords

Deep neural networks
bottleneck features
probability density
statistical normalisation

Access to Document

10.1109/ICASSP.2019.8683330

ICASSP19_DNN_StatisticalAccepted author manuscript, 434 KB

https://ieeexplore.ieee.org/abstract/document/8683330

SpeechWave
Renals, S. & Bell, P.
EPSRC
1/03/18 → 21/05/22
Project: Research

Cite this

@inproceedings{0d5be74be4884aac9645d073212b2e32,

title = "On the Usefulness of Statistical Normalisation of Bottleneck Features for Speech Recognition",

abstract = "DNNs play a major role in the state-of-the-art ASR systems. They can be used for extracting features and building probabilistic models for acoustic and language modelling. Despite their huge practical success, the level of theoretical understanding has remained shallow. This paper investigates DNNs from a statistical standpoint. In particular, the effect of activation functions on the distribution of the pre-activations and activations is investigated and discussed from both analytic and empirical viewpoints. This study, among others, shows that the pre-activation density in the bottleneck layer can be well fitted with a diagonal GMM with a few Gaussians and how and why the ReLU activation function promotes sparsity. Motivated by the statistical properties of the pre-activations, the usefulness of statistical normalisation of bottleneck features was also investigated. To this end, methods such as mean(-variance) normalisation, Gaussianisation, and histogram equalisation (HEQ) were employed and up to 2% (absolute) WER reduction achieved in the Aurora-4 task.",

keywords = "Deep neural networks, bottleneck features, probability density, statistical normalisation",

author = "Erfan Loweimi and Peter Bell and Steve Renals",

year = "2019",

month = apr,

day = "17",

doi = "10.1109/ICASSP.2019.8683330",

language = "English",

isbn = "978-1-4799-8132-8",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

pages = "3862--3866",

booktitle = "ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)",

address = "United States",

note = "44th International Conference on Acoustics, Speech, and Signal Processing : Signal Processing: Empowering Science and Technology for Humankind, ICASSP 2019 ; Conference date: 12-05-2019 Through 17-05-2019",

url = "https://2019.ieeeicassp.org/",

}

Loweimi, E, Bell, P & Renals, S 2019, On the Usefulness of Statistical Normalisation of Bottleneck Features for Speech Recognition. in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Institute of Electrical and Electronics Engineers (IEEE), Brighton, United Kingdom, pp. 3862-3866, 44th International Conference on Acoustics, Speech, and Signal Processing, Brighton , United Kingdom, 12/05/19. https://doi.org/10.1109/ICASSP.2019.8683330

On the Usefulness of Statistical Normalisation of Bottleneck Features for Speech Recognition. / Loweimi, Erfan; Bell, Peter ; Renals, Steve.

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Brighton, United Kingdom : Institute of Electrical and Electronics Engineers (IEEE), 2019. p. 3862-3866.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - On the Usefulness of Statistical Normalisation of Bottleneck Features for Speech Recognition

AU - Loweimi, Erfan

AU - Bell, Peter

AU - Renals, Steve

N1 - Conference code: 44

PY - 2019/4/17

Y1 - 2019/4/17

N2 - DNNs play a major role in the state-of-the-art ASR systems. They can be used for extracting features and building probabilistic models for acoustic and language modelling. Despite their huge practical success, the level of theoretical understanding has remained shallow. This paper investigates DNNs from a statistical standpoint. In particular, the effect of activation functions on the distribution of the pre-activations and activations is investigated and discussed from both analytic and empirical viewpoints. This study, among others, shows that the pre-activation density in the bottleneck layer can be well fitted with a diagonal GMM with a few Gaussians and how and why the ReLU activation function promotes sparsity. Motivated by the statistical properties of the pre-activations, the usefulness of statistical normalisation of bottleneck features was also investigated. To this end, methods such as mean(-variance) normalisation, Gaussianisation, and histogram equalisation (HEQ) were employed and up to 2% (absolute) WER reduction achieved in the Aurora-4 task.

AB - DNNs play a major role in the state-of-the-art ASR systems. They can be used for extracting features and building probabilistic models for acoustic and language modelling. Despite their huge practical success, the level of theoretical understanding has remained shallow. This paper investigates DNNs from a statistical standpoint. In particular, the effect of activation functions on the distribution of the pre-activations and activations is investigated and discussed from both analytic and empirical viewpoints. This study, among others, shows that the pre-activation density in the bottleneck layer can be well fitted with a diagonal GMM with a few Gaussians and how and why the ReLU activation function promotes sparsity. Motivated by the statistical properties of the pre-activations, the usefulness of statistical normalisation of bottleneck features was also investigated. To this end, methods such as mean(-variance) normalisation, Gaussianisation, and histogram equalisation (HEQ) were employed and up to 2% (absolute) WER reduction achieved in the Aurora-4 task.

KW - Deep neural networks

KW - bottleneck features

KW - probability density

KW - statistical normalisation

U2 - 10.1109/ICASSP.2019.8683330

DO - 10.1109/ICASSP.2019.8683330

M3 - Conference contribution

SN - 978-1-4799-8132-8

SP - 3862

EP - 3866

BT - ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

PB - Institute of Electrical and Electronics Engineers (IEEE)

CY - Brighton, United Kingdom

T2 - 44th International Conference on Acoustics, Speech, and Signal Processing

Y2 - 12 May 2019 through 17 May 2019

ER -

Loweimi E, Bell P , Renals S. On the Usefulness of Statistical Normalisation of Bottleneck Features for Speech Recognition. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Brighton, United Kingdom: Institute of Electrical and Electronics Engineers (IEEE). 2019. p. 3862-3866 Epub 2019 Apr 17. doi: 10.1109/ICASSP.2019.8683330

On the Usefulness of Statistical Normalisation of Bottleneck Features for Speech Recognition

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Projects

SpeechWave

Cite this