Continuous space discriminative language modeling

Puyang Xu; Sanjeev Khudanpur; Maider Lehr; Emily Tucker Prud'hommeaux; Nathan Glenn; Damianos Karakos; Brian Roark; Kenji Sagae; Murat Saraclar; Izhak Shafran; Daniel M. Bikel; Chris Callison-Burch; Yuan Cao; Keith B. Hall; Eva Hasler; Philipp Koehn; Adam Lopez; Matt Post; Darcey Riley

doi:10.1109/ICASSP.2012.6288332

Continuous space discriminative language modeling

Puyang Xu
, Sanjeev Khudanpur
, Maider Lehr
, Emily Tucker Prud'hommeaux
, Nathan Glenn
, Damianos Karakos
, Brian Roark
, Kenji Sagae
, Murat Saraclar
, Izhak Shafran
, Daniel M. Bikel
, Chris Callison-Burch
, Yuan Cao
, Keith B. Hall
, Eva Hasler
, Philipp Koehn
, Adam Lopez
, Matt Post
, Darcey Riley

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Discriminative language modeling is a structured classification problem. Log-linear models have been previously used to address this problem. In this paper, the standard dot-product feature representation used in log-linear models is replaced by a non-linear function parameterized by a neural network. Embeddings are learned for each word and features are extracted automatically through the use of convolutional layers. Experimental results show that as a stand-alone model the continuous space model yields significantly lower word error rate (1% absolute), while having a much more compact parameterization (60%-90% smaller). If the baseline scores are combined, our approach performs equally well.

Original language	English
Title of host publication	2012 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2012, Kyoto, Japan, March 25-30, 2012
Publisher	Institute of Electrical and Electronics Engineers
Pages	2129-2132
Number of pages	4
ISBN (Electronic)	978-1-4673-0044-5
ISBN (Print)	978-1-4673-0045-2
DOIs	https://doi.org/10.1109/ICASSP.2012.6288332
Publication status	Published - 2012

Access to Document

10.1109/ICASSP.2012.6288332

Khundanpur Lehr ET AL 2012 Continuous space discriminative language modelingAccepted author manuscript, 347 KB

http://dx.doi.org/10.1109/ICASSP.2012.6288332

Cite this

Xu, P., Khudanpur, S., Lehr, M., Prud'hommeaux, E. T., Glenn, N., Karakos, D., Roark, B., Sagae, K., Saraclar, M., Shafran, I., Bikel, D. M., Callison-Burch, C., Cao, Y., Hall, K. B., Hasler, E., Koehn, P., Lopez, A., Post, M., & Riley, D. (2012). Continuous space discriminative language modeling. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2012, Kyoto, Japan, March 25-30, 2012 (pp. 2129-2132). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICASSP.2012.6288332

@inproceedings{0f85515f1dd248999edd1b5dfa8d6094,

title = "Continuous space discriminative language modeling",

abstract = "Discriminative language modeling is a structured classification problem. Log-linear models have been previously used to address this problem. In this paper, the standard dot-product feature representation used in log-linear models is replaced by a non-linear function parameterized by a neural network. Embeddings are learned for each word and features are extracted automatically through the use of convolutional layers. Experimental results show that as a stand-alone model the continuous space model yields significantly lower word error rate (1\% absolute), while having a much more compact parameterization (60\%-90\% smaller). If the baseline scores are combined, our approach performs equally well.",

author = "Puyang Xu and Sanjeev Khudanpur and Maider Lehr and Prud'hommeaux, \{Emily Tucker\} and Nathan Glenn and Damianos Karakos and Brian Roark and Kenji Sagae and Murat Saraclar and Izhak Shafran and Bikel, \{Daniel M.\} and Chris Callison-Burch and Yuan Cao and Hall, \{Keith B.\} and Eva Hasler and Philipp Koehn and Adam Lopez and Matt Post and Darcey Riley",

year = "2012",

doi = "10.1109/ICASSP.2012.6288332",

language = "English",

isbn = "978-1-4673-0045-2",

pages = "2129--2132",

booktitle = "2012 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2012, Kyoto, Japan, March 25-30, 2012",

publisher = "Institute of Electrical and Electronics Engineers",

address = "United States",

}

Xu, P, Khudanpur, S, Lehr, M, Prud'hommeaux, ET, Glenn, N, Karakos, D, Roark, B, Sagae, K, Saraclar, M, Shafran, I, Bikel, DM, Callison-Burch, C, Cao, Y, Hall, KB, Hasler, E, Koehn, P, Lopez, A, Post, M & Riley, D 2012, Continuous space discriminative language modeling. in 2012 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2012, Kyoto, Japan, March 25-30, 2012. Institute of Electrical and Electronics Engineers, pp. 2129-2132. https://doi.org/10.1109/ICASSP.2012.6288332

Continuous space discriminative language modeling. / Xu, Puyang; Khudanpur, Sanjeev; Lehr, Maider et al.
2012 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2012, Kyoto, Japan, March 25-30, 2012. Institute of Electrical and Electronics Engineers, 2012. p. 2129-2132.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Continuous space discriminative language modeling

AU - Xu, Puyang

AU - Khudanpur, Sanjeev

AU - Lehr, Maider

AU - Prud'hommeaux, Emily Tucker

AU - Glenn, Nathan

AU - Karakos, Damianos

AU - Roark, Brian

AU - Sagae, Kenji

AU - Saraclar, Murat

AU - Shafran, Izhak

AU - Bikel, Daniel M.

AU - Callison-Burch, Chris

AU - Cao, Yuan

AU - Hall, Keith B.

AU - Hasler, Eva

AU - Koehn, Philipp

AU - Lopez, Adam

AU - Post, Matt

AU - Riley, Darcey

PY - 2012

Y1 - 2012

N2 - Discriminative language modeling is a structured classification problem. Log-linear models have been previously used to address this problem. In this paper, the standard dot-product feature representation used in log-linear models is replaced by a non-linear function parameterized by a neural network. Embeddings are learned for each word and features are extracted automatically through the use of convolutional layers. Experimental results show that as a stand-alone model the continuous space model yields significantly lower word error rate (1% absolute), while having a much more compact parameterization (60%-90% smaller). If the baseline scores are combined, our approach performs equally well.

AB - Discriminative language modeling is a structured classification problem. Log-linear models have been previously used to address this problem. In this paper, the standard dot-product feature representation used in log-linear models is replaced by a non-linear function parameterized by a neural network. Embeddings are learned for each word and features are extracted automatically through the use of convolutional layers. Experimental results show that as a stand-alone model the continuous space model yields significantly lower word error rate (1% absolute), while having a much more compact parameterization (60%-90% smaller). If the baseline scores are combined, our approach performs equally well.

U2 - 10.1109/ICASSP.2012.6288332

DO - 10.1109/ICASSP.2012.6288332

M3 - Conference contribution

SN - 978-1-4673-0045-2

SP - 2129

EP - 2132

BT - 2012 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2012, Kyoto, Japan, March 25-30, 2012

PB - Institute of Electrical and Electronics Engineers

ER -

Continuous space discriminative language modeling

Abstract

Access to Document

Fingerprint

Cite this