Edinburgh Research Explorer

Knowledge Distillation for Small-footprint Highway Networks

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Related Edinburgh Organisations

Open Access permissions

Open

Documents

http://ieeexplore.ieee.org/document/7953072/
Original languageEnglish
Title of host publication2017 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP 2017)
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages4280-4284
Number of pages5
ISBN (Electronic)978-1-5090-4117-6
DOIs
Publication statusPublished - 19 Jun 2017
Event42nd IEEE International Conference on Acoustics, Speech and Signal Processing - New Orleans, United States
Duration: 5 Mar 20179 Mar 2017
http://www.ieee-icassp2017.org/

Conference

Conference42nd IEEE International Conference on Acoustics, Speech and Signal Processing
Abbreviated titleICASSP 2017
CountryUnited States
CityNew Orleans
Period5/03/179/03/17
Internet address

Abstract

Deep learning has significantly advanced state-of-the-art of speech recognition in the past few years. However, compared to conventional Gaussian mixture acoustic models, neural network models are usually much larger, and are therefore not very deployable in embedded devices. Previously, we investigated a compact highway deep neural network (HDNN) for acoustic modelling, which is a type of depth-gated feedforward neural network. We have shown that HDNN-based acoustic models can achieve comparable recognition accuracy with much smaller number of model parameters compared to plain deep neural network (DNN) acoustic models. In this paper, we push the boundary further by leveraging on the knowledge distillation technique that is also known as teacher-student training, i.e., we train the compact HDNN model with the supervision of a high accuracy cumbersome model. Furthermore, we also investigate sequence training and adaptation in the context of teacher-student training. Our experiments were performed on the AMI meeting speech recognition corpus. With this technique, we significantly improved the recognition accuracy of the HDNN acoustic model with less than 0.8 million parameters, and narrowed the gap between this model and the plain DNN with 30 million parameters.

Event

42nd IEEE International Conference on Acoustics, Speech and Signal Processing

5/03/179/03/17

New Orleans, United States

Event: Conference

Download statistics

No data available

ID: 31853044