Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models

Gongbo Tang; Rico Sennrich; Joakim Nivre

doi:10.26615/978-954-452-056-4_136

Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models

Gongbo Tang, Rico Sennrich, Joakim Nivre

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

In this paper, we try to understand neural machine translation (NMT) via simplifying NMT architectures and training encoder-free NMT models. In an encoderfree model, the sums of word embeddings and positional embeddings represent the source. The decoder is a standard Transformer or recurrent neural network that directly attends to embeddings via attention mechanisms. Experimental results show (1) that the attention mechanism in encoder-free models acts as a strong feature extractor, (2) that the word embeddings in encoder-free models are competitive to those in conventional models, (3) that non-contextualized source representations lead to a big performance drop, and (4) that encoder-free models have different effects on alignment quality for German→English and Chinese→English.

Original language	English
Title of host publication	Recent Advances in Natural Processing 2019
Subtitle of host publication	RANLP 2019: Natural Language Processingin a Deep Learning World
Publisher	INCOMA Ltd.
Pages	1186-1193
Number of pages	8
ISBN (Electronic)	978-954-452-056-4
ISBN (Print)	978-954-452-055-7
DOIs	https://doi.org/10.26615/978-954-452-056-4_136
Publication status	Published - 30 Sept 2019
Event	Recent Advances in Natural Language Processing (RANLP 2019) - Cherno More Hotel, Varna, Bulgaria Duration: 2 Sept 2019 → 4 Sept 2019 http://lml.bas.bg/ranlp2019/start.php

Publication series

Name	Natural Language Processing in a Deep Learning World
Publisher	INCOMA Ltd.
ISSN (Print)	1313-8502
ISSN (Electronic)	2603-2813

Conference

Conference	Recent Advances in Natural Language Processing (RANLP 2019)
Abbreviated title	RANLP 2019
Country/Territory	Bulgaria
City	Varna
Period	2/09/19 → 4/09/19
Internet address	http://lml.bas.bg/ranlp2019/start.php

Access to Document

10.26615/978-954-452-056-4_136Licence: All Rights Reserved

Understanding Neural Machine_TANG_DoA060719_AFVAccepted author manuscript, 398 KB

https://acl-bg.org/proceedings/2019/RANLP%202019/index.htmlLicence: All Rights Reserved

Cite this

Tang, G., Sennrich, R., & Nivre, J. (2019). Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models. In Recent Advances in Natural Processing 2019: RANLP 2019: Natural Language Processingin a Deep Learning World (pp. 1186-1193). (Natural Language Processing in a Deep Learning World). INCOMA Ltd.. https://doi.org/10.26615/978-954-452-056-4_136

@inproceedings{b15de884a3dc4f59ac1ee6449fa714f5,

title = "Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models",

abstract = "In this paper, we try to understand neural machine translation (NMT) via simplifying NMT architectures and training encoder-free NMT models. In an encoderfree model, the sums of word embeddings and positional embeddings represent the source. The decoder is a standard Transformer or recurrent neural network that directly attends to embeddings via attention mechanisms. Experimental results show (1) that the attention mechanism in encoder-free models acts as a strong feature extractor, (2) that the word embeddings in encoder-free models are competitive to those in conventional models, (3) that non-contextualized source representations lead to a big performance drop, and (4) that encoder-free models have different effects on alignment quality for German→English and Chinese→English. ",

author = "Gongbo Tang and Rico Sennrich and Joakim Nivre",

year = "2019",

month = sep,

day = "30",

doi = "10.26615/978-954-452-056-4_136",

language = "English",

isbn = "978-954-452-055-7",

series = "Natural Language Processing in a Deep Learning World",

publisher = "INCOMA Ltd.",

pages = "1186--1193",

booktitle = "Recent Advances in Natural Processing 2019",

note = "Recent Advances in Natural Language Processing (RANLP 2019), RANLP 2019 ; Conference date: 02-09-2019 Through 04-09-2019",

url = "http://lml.bas.bg/ranlp2019/start.php",

}

Tang, G, Sennrich, R & Nivre, J 2019, Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models. in Recent Advances in Natural Processing 2019: RANLP 2019: Natural Language Processingin a Deep Learning World. Natural Language Processing in a Deep Learning World, INCOMA Ltd., pp. 1186-1193, Recent Advances in Natural Language Processing (RANLP 2019), Varna, Bulgaria, 2/09/19. https://doi.org/10.26615/978-954-452-056-4_136

Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models. / Tang, Gongbo; Sennrich, Rico; Nivre, Joakim.
Recent Advances in Natural Processing 2019: RANLP 2019: Natural Language Processingin a Deep Learning World. INCOMA Ltd., 2019. p. 1186-1193 (Natural Language Processing in a Deep Learning World).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models

AU - Tang, Gongbo

AU - Sennrich, Rico

AU - Nivre, Joakim

PY - 2019/9/30

Y1 - 2019/9/30

N2 - In this paper, we try to understand neural machine translation (NMT) via simplifying NMT architectures and training encoder-free NMT models. In an encoderfree model, the sums of word embeddings and positional embeddings represent the source. The decoder is a standard Transformer or recurrent neural network that directly attends to embeddings via attention mechanisms. Experimental results show (1) that the attention mechanism in encoder-free models acts as a strong feature extractor, (2) that the word embeddings in encoder-free models are competitive to those in conventional models, (3) that non-contextualized source representations lead to a big performance drop, and (4) that encoder-free models have different effects on alignment quality for German→English and Chinese→English.

AB - In this paper, we try to understand neural machine translation (NMT) via simplifying NMT architectures and training encoder-free NMT models. In an encoderfree model, the sums of word embeddings and positional embeddings represent the source. The decoder is a standard Transformer or recurrent neural network that directly attends to embeddings via attention mechanisms. Experimental results show (1) that the attention mechanism in encoder-free models acts as a strong feature extractor, (2) that the word embeddings in encoder-free models are competitive to those in conventional models, (3) that non-contextualized source representations lead to a big performance drop, and (4) that encoder-free models have different effects on alignment quality for German→English and Chinese→English.

U2 - 10.26615/978-954-452-056-4_136

DO - 10.26615/978-954-452-056-4_136

M3 - Conference contribution

SN - 978-954-452-055-7

T3 - Natural Language Processing in a Deep Learning World

SP - 1186

EP - 1193

BT - Recent Advances in Natural Processing 2019

PB - INCOMA Ltd.

T2 - Recent Advances in Natural Language Processing (RANLP 2019)

Y2 - 2 September 2019 through 4 September 2019

ER -

Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models

Abstract / Description of output

Publication series

Conference

Access to Document

Fingerprint

Cite this