Edinburgh Research Explorer

Deep Architectures for Neural Machine Translation

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Original languageEnglish
Title of host publicationProceedings of the Second Conference on Machine Translation, Volume 1: Research Papers
Place of PublicationCopenhagen, Denmark
PublisherAssociation for Computational Linguistics
Pages99-107
Number of pages9
DOIs
Publication statusPublished - 8 Sep 2017
EventProceedings of the Second Conference on Machine Translation - Copenhagen, Denmark
Duration: 7 Sep 20178 Sep 2017
http://www.statmt.org/wmt17/

Conference

ConferenceProceedings of the Second Conference on Machine Translation
Abbreviated titleWMT17
CountryDenmark
CityCopenhagen
Period7/09/178/09/17
Internet address

Abstract

It has been shown that increasing model depth improves the quality of neural machine translation. However, different architectural variants to increase model depth have been proposed, and so far, there has been no thorough comparative study. In this work, we describe and evaluate several existing approaches to introduce depth in neural machine translation. Additionally, we explore novel architectural variants, including deep transition RNNs, and we vary how attention is used in the deep decoder. We introduce a novel "BiDeep" RNN architecture that combines deep transition RNNs and stacked RNNs. Our evaluation is carried out on the English to German WMT news translation dataset, using a single-GPU machine for both training and inference. We find that several of our proposed architectures improve upon existing approaches in terms of speed and translation quality. We obtain best improvements with a BiDeep RNN of combined depth 8, obtaining an average improvement of 1.5 BLEU over a strong shallow baseline. We release our code for ease of adoption.

Event

Proceedings of the Second Conference on Machine Translation

7/09/178/09/17

Copenhagen, Denmark

Event: Conference

Download statistics

No data available

ID: 40221224