Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers

Shucong Zhang; Cong-Thanh Do; Rama Doddipatla; Erfan Loweimi; Peter Bell; Steve Renals

Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers

Shucong Zhang, Cong-Thanh Do, Rama Doddipatla, Erfan Loweimi, Peter Bell, Steve Renals

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Although the lower layers of a deep neural network learn features which are transferable across datasets, these layers are not transferable within the same dataset. That is, in general, freezing the trained feature extractor (the lower layers) and retraining the classifier (the upper layers) on the same datasetleads to worse performance. In this paper, for the first time, we show that the frozen classifier is transferable within the same dataset. We develop a novel top-down training method which can be viewed as an algorithm for searching for high-quality classifiers. We tested this method on automatic speech recognition (ASR) tasks and language modelling tasks. The proposed method consistently improves recurrent neural network ASR models on Wall Street Journal, self-attention ASR models on Switchboard, and AWD-LSTM language models on WikiText-2.

Original language	English
Title of host publication	2021 IEEE International Conference on Acoustics, Speech and Signal Processing
Publisher	Institute of Electrical and Electronics Engineers (IEEE)
Number of pages	5
Publication status	Accepted/In press - 30 Jan 2021
Event	46th IEEE International Conference on Acoustics, Speech and Signal Processing - Toronto, Canada Duration: 6 Jun 2021 → 11 Jun 2021 https://2021.ieeeicassp.org/

Conference

Conference	46th IEEE International Conference on Acoustics, Speech and Signal Processing
Abbreviated title	ICASSP 2021
Country	Canada
City	Toronto
Period	6/06/21 → 11/06/21
Internet address	https://2021.ieeeicassp.org/

Keywords

top-down training
layer-wise training
general classifier
speech recognition
language model

Embargoed Document

Train Your Classifier_ZHANG_DOA30012021_AFV
Accepted author manuscript, 188 KB
Licence: All Rights Reserved
Embargo ends: 11/06/21

Fingerprint Dive into the research topics of 'Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers'. Together they form a unique fingerprint.

Cite this

@inproceedings{a77d67240f804c9c8f41987083720384,

title = "Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers",

abstract = "Although the lower layers of a deep neural network learn features which are transferable across datasets, these layers are not transferable within the same dataset. That is, in general, freezing the trained feature extractor (the lower layers) and retraining the classifier (the upper layers) on the same datasetleads to worse performance. In this paper, for the first time, we show that the frozen classifier is transferable within the same dataset. We develop a novel top-down training method which can be viewed as an algorithm for searching for high-quality classifiers. We tested this method on automatic speech recognition (ASR) tasks and language modelling tasks. The proposed method consistently improves recurrent neural network ASR models on Wall Street Journal, self-attention ASR models on Switchboard, and AWD-LSTM language models on WikiText-2.",

keywords = "top-down training, layer-wise training, general classifier, speech recognition, language model",

author = "Shucong Zhang and Cong-Thanh Do and Rama Doddipatla and Erfan Loweimi and Peter Bell and Steve Renals",

year = "2021",

month = jan,

day = "30",

language = "English",

booktitle = "2021 IEEE International Conference on Acoustics, Speech and Signal Processing",

publisher = "Institute of Electrical and Electronics Engineers (IEEE)",

address = "United States",

note = "46th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2021 ; Conference date: 06-06-2021 Through 11-06-2021",

url = "https://2021.ieeeicassp.org/",

}

Zhang, S, Do, C-T, Doddipatla, R, Loweimi, E , Bell, P & Renals, S 2021, Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers. in 2021 IEEE International Conference on Acoustics, Speech and Signal Processing. Institute of Electrical and Electronics Engineers (IEEE), 46th IEEE International Conference on Acoustics, Speech and Signal Processing, Toronto, Canada, 6/06/21.

Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers. / Zhang, Shucong; Do, Cong-Thanh; Doddipatla, Rama; Loweimi, Erfan ; Bell, Peter ; Renals, Steve.

2021 IEEE International Conference on Acoustics, Speech and Signal Processing. Institute of Electrical and Electronics Engineers (IEEE), 2021.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Train your classifier first: Cascade Neural Networks Training from upper layers to lower layers

AU - Zhang, Shucong

AU - Do, Cong-Thanh

AU - Doddipatla, Rama

AU - Loweimi, Erfan

AU - Bell, Peter

AU - Renals, Steve

PY - 2021/1/30

Y1 - 2021/1/30

N2 - Although the lower layers of a deep neural network learn features which are transferable across datasets, these layers are not transferable within the same dataset. That is, in general, freezing the trained feature extractor (the lower layers) and retraining the classifier (the upper layers) on the same datasetleads to worse performance. In this paper, for the first time, we show that the frozen classifier is transferable within the same dataset. We develop a novel top-down training method which can be viewed as an algorithm for searching for high-quality classifiers. We tested this method on automatic speech recognition (ASR) tasks and language modelling tasks. The proposed method consistently improves recurrent neural network ASR models on Wall Street Journal, self-attention ASR models on Switchboard, and AWD-LSTM language models on WikiText-2.

AB - Although the lower layers of a deep neural network learn features which are transferable across datasets, these layers are not transferable within the same dataset. That is, in general, freezing the trained feature extractor (the lower layers) and retraining the classifier (the upper layers) on the same datasetleads to worse performance. In this paper, for the first time, we show that the frozen classifier is transferable within the same dataset. We develop a novel top-down training method which can be viewed as an algorithm for searching for high-quality classifiers. We tested this method on automatic speech recognition (ASR) tasks and language modelling tasks. The proposed method consistently improves recurrent neural network ASR models on Wall Street Journal, self-attention ASR models on Switchboard, and AWD-LSTM language models on WikiText-2.

KW - top-down training

KW - layer-wise training

KW - general classifier

KW - speech recognition

KW - language model

M3 - Conference contribution

BT - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing

PB - Institute of Electrical and Electronics Engineers (IEEE)

T2 - 46th IEEE International Conference on Acoustics, Speech and Signal Processing

Y2 - 6 June 2021 through 11 June 2021

ER -