Abstract
Recently, it has been argued that encoder-decoder models can be made more interpretable by replacing the softmax function in the attention with its sparse variants. In this work, we introduce a novel, simple method for achieving sparsity in attention: we replace the softmax activation with a ReLU, and show that sparsity naturally emerges from such a formulation. Training stability is achieved with layer normalization with either a specialized initialization or an additional gating function. Our model, which we call Rectified Linear Attention (ReLA), is easy to implement and more efficient than previously proposed sparse attention mechanisms. We apply ReLA to the Transformer and conduct experiments on five machine translation tasks. ReLA achieves translation performance comparable to several strong baselines, with training and decoding speed similar to that of the vanilla attention. Our analysis shows that ReLA delivers high sparsity rate and head diversity, and the induced cross attention achieves better accuracy with respect to source-target word alignment than recent sparsified softmax-based models. Intriguingly, ReLA heads also learn to attend to nothing (i.e. `switch off') for some queries, which is not possible with sparsified softmax alternatives.
Original language | English |
---|---|
Title of host publication | Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing |
Place of Publication | Online and Punta Cana, Dominican Republic |
Publisher | Association for Computational Linguistics |
Pages | 6507-6520 |
Number of pages | 14 |
ISBN (Electronic) | 978-1-955917-09-4 |
Publication status | Published - 7 Nov 2021 |
Event | 2021 Conference on Empirical Methods in Natural Language Processing - Punta Cana, Dominican Republic Duration: 7 Nov 2021 → 11 Nov 2021 https://2021.emnlp.org/ |
Conference
Conference | 2021 Conference on Empirical Methods in Natural Language Processing |
---|---|
Abbreviated title | EMNLP 2021 |
Country/Territory | Dominican Republic |
City | Punta Cana |
Period | 7/11/21 → 11/11/21 |
Internet address |