BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

Asa Cooper Stickland; Iain Murray

BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

Asa Cooper Stickland, Iain Murray

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

Multi-task learning shares information between related tasks, sometimes reducing the number of parameters required. State-of-the-art results across multiple natural language understanding tasks in the GLUE benchmark have previously used transfer from a single large task: unsupervised pre-training with BERT, where a separate BERT model was fine-tuned for each task. We explore multi-task approaches that share a single BERT model with a small number of additional task-specific parameters. Using new adaptation modules, PALs or ‘projected attention layers’, we match the performance of separately finetuned models on the GLUE benchmark with ≈7 times fewer parameters, and obtain state-of-theart results on the Recognizing Textual Entailment dataset.

Original language	English
Title of host publication	Proceedings of the 36th International Conference on Machine Learning (ICML)
Editors	Kamalika Chaudhuri, Ruslan Salakhutdinov
Place of Publication	Long Beach, USA
Publisher	PMLR
Pages	5986-5995
Number of pages	12
Volume	97
Publication status	E-pub ahead of print - 3 Jul 2019
Event	Thirty-sixth International Conference on Machine Learning - Long Beach Convention Center, Long Beach, United States Duration: 9 Jun 2019 → 15 Jun 2019 Conference number: 36 https://icml.cc/Conferences/2019

Publication series

Name	Proceedings of Machine Learning Research
Publisher	PMLR
Volume	97
ISSN (Electronic)	2640-3498

Conference

Conference	Thirty-sixth International Conference on Machine Learning
Abbreviated title	ICML 2019
Country/Territory	United States
City	Long Beach
Period	9/06/19 → 15/06/19
Internet address	https://icml.cc/Conferences/2019

Access to Document

BERT and PALs_COOPER_STICKLAND_DoA210419_AFVAccepted author manuscript, 167 KB

Cite this

Cooper Stickland, A., & Murray, I. (2019). BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning. In K. Chaudhuri, & R. Salakhutdinov (Eds.), Proceedings of the 36th International Conference on Machine Learning (ICML) (Vol. 97, pp. 5986-5995). (Proceedings of Machine Learning Research; Vol. 97). PMLR. Advance online publication. http://proceedings.mlr.press/v97/stickland19a.html

@inproceedings{93420d8b127b4f4cb012d502fa20a088,

title = "BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning",

abstract = "Multi-task learning shares information between related tasks, sometimes reducing the number of parameters required. State-of-the-art results across multiple natural language understanding tasks in the GLUE benchmark have previously used transfer from a single large task: unsupervised pre-training with BERT, where a separate BERT model was fine-tuned for each task. We explore multi-task approaches that share a single BERT model with a small number of additional task-specific parameters. Using new adaptation modules, PALs or {\textquoteleft}projected attention layers{\textquoteright}, we match the performance of separately finetuned models on the GLUE benchmark with ≈7 times fewer parameters, and obtain state-of-theart results on the Recognizing Textual Entailment dataset.",

author = "{Cooper Stickland}, Asa and Iain Murray",

year = "2019",

month = jul,

day = "3",

language = "English",

volume = "97",

series = "Proceedings of Machine Learning Research",

publisher = "PMLR",

pages = "5986--5995",

editor = "Kamalika Chaudhuri and Ruslan Salakhutdinov",

booktitle = "Proceedings of the 36th International Conference on Machine Learning (ICML)",

note = "Thirty-sixth International Conference on Machine Learning, ICML 2019 ; Conference date: 09-06-2019 Through 15-06-2019",

url = "https://icml.cc/Conferences/2019",

}

Cooper Stickland, A & Murray, I 2019, BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning. in K Chaudhuri & R Salakhutdinov (eds), Proceedings of the 36th International Conference on Machine Learning (ICML). vol. 97, Proceedings of Machine Learning Research, vol. 97, PMLR, Long Beach, USA, pp. 5986-5995, Thirty-sixth International Conference on Machine Learning, Long Beach, California, United States, 9/06/19. <http://proceedings.mlr.press/v97/stickland19a.html>

BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning. / Cooper Stickland, Asa; Murray, Iain.
Proceedings of the 36th International Conference on Machine Learning (ICML). ed. / Kamalika Chaudhuri; Ruslan Salakhutdinov. Vol. 97 Long Beach, USA: PMLR, 2019. p. 5986-5995 (Proceedings of Machine Learning Research; Vol. 97).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

AU - Cooper Stickland, Asa

AU - Murray, Iain

N1 - Conference code: 36

PY - 2019/7/3

Y1 - 2019/7/3

N2 - Multi-task learning shares information between related tasks, sometimes reducing the number of parameters required. State-of-the-art results across multiple natural language understanding tasks in the GLUE benchmark have previously used transfer from a single large task: unsupervised pre-training with BERT, where a separate BERT model was fine-tuned for each task. We explore multi-task approaches that share a single BERT model with a small number of additional task-specific parameters. Using new adaptation modules, PALs or ‘projected attention layers’, we match the performance of separately finetuned models on the GLUE benchmark with ≈7 times fewer parameters, and obtain state-of-theart results on the Recognizing Textual Entailment dataset.

AB - Multi-task learning shares information between related tasks, sometimes reducing the number of parameters required. State-of-the-art results across multiple natural language understanding tasks in the GLUE benchmark have previously used transfer from a single large task: unsupervised pre-training with BERT, where a separate BERT model was fine-tuned for each task. We explore multi-task approaches that share a single BERT model with a small number of additional task-specific parameters. Using new adaptation modules, PALs or ‘projected attention layers’, we match the performance of separately finetuned models on the GLUE benchmark with ≈7 times fewer parameters, and obtain state-of-theart results on the Recognizing Textual Entailment dataset.

UR - http://proceedings.mlr.press/v97/

M3 - Conference contribution

VL - 97

T3 - Proceedings of Machine Learning Research

SP - 5986

EP - 5995

BT - Proceedings of the 36th International Conference on Machine Learning (ICML)

A2 - Chaudhuri, Kamalika

A2 - Salakhutdinov, Ruslan

PB - PMLR

CY - Long Beach, USA

T2 - Thirty-sixth International Conference on Machine Learning

Y2 - 9 June 2019 through 15 June 2019

ER -

BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

Abstract / Description of output

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this