Distributionally Robust Recurrent Decoders with Random Network Distillation

Antonio Valerio Miceli Barone; Alexandra Birch; Rico Sennrich

doi:10.18653/v1/2022.repl4nlp-1.1

Distributionally Robust Recurrent Decoders with Random Network Distillation

Antonio Valerio Miceli Barone, Alexandra Birch, Rico Sennrich

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Neural machine learning models can successfully model language that is similar to their training distribution, but they are highly susceptible to degradation under distribution shift, which occurs in many practical applications when processing out-of-domain (OOD) text. This has been attributed to ``shortcut learning'''':'' relying on weak correlations over arbitrary large contexts. We propose a method based on OOD detection with Random Network Distillation to allow an autoregressive language model to automatically disregard OOD context during inference, smoothly transitioning towards a less expressive but more robust model as the data becomes more OOD, while retaining its full context capability when operating in-distribution. We apply our method to a GRU architecture, demonstrating improvements on multiple language modeling (LM) datasets.

Original language	English
Title of host publication	Proceedings of the 7th Workshop on Representation Learning for NLP
Editors	Spandana Gella, He He, Bodhisattwa Prasad Majumdar, Burcu Can, Eleonora Giunchiglia, Samuel Cahyawijaya, Sewon Min, Maximillian Mozes, Xiang Lorraine Li, Isabelle Augenstein, Anna Rogers, Kyunghyun Cho, Edward Grefenstette, Laura Rimell, Chris Dyer
Place of Publication	Dublin, Ireland
Publisher	Association for Computational Linguistics
Pages	1-8
Number of pages	8
ISBN (Electronic)	978-1-955917-48-3
DOIs	https://doi.org/10.18653/v1/2022.repl4nlp-1.1
Publication status	Published - 3 Jun 2022
Event	The 7th Workshop on Representation Learning for NLP - Dublin, Ireland Duration: 26 May 2022 → 26 May 2022 Conference number: 7 https://sites.google.com/view/repl4nlp2022/home

Workshop

Workshop	The 7th Workshop on Representation Learning for NLP
Abbreviated title	Repl4NLP 2022
Country/Territory	Ireland
City	Dublin
Period	26/05/22 → 26/05/22
Internet address	https://sites.google.com/view/repl4nlp2022/home

Access to Document

10.18653/v1/2022.repl4nlp-1.1Licence: Creative Commons: Attribution (CC-BY)

Distributionally Robust_BARONE_DOA26032022_VOR_CC-BYFinal published version, 229 KBLicence: Creative Commons: Attribution (CC-BY)

https://aclanthology.org/2022.repl4nlp-1.1Licence: Creative Commons: Attribution (CC-BY)

Global Under-Resourced MEdia Translation
Birch-Mayne, A. (Principal Investigator) & Haddow, B. (Co-investigator)
EU government bodies
1/01/19 → 30/06/22
Project: Research
MTStretch: Low-resource Machine Translation
Birch-Mayne, A. (Principal Investigator)
EPSRC
29/06/18 → 28/12/21
Project: Research
BroadSem-Induction of Broad-Coverage Semantic Parsers
Titov, I. (Principal Investigator)
EU government bodies
1/05/17 → 30/04/22
Project: Research

Cite this

Valerio Miceli Barone, A., Birch, A., & Sennrich, R. (2022). Distributionally Robust Recurrent Decoders with Random Network Distillation. In S. Gella, H. He, B. P. Majumdar, B. Can, E. Giunchiglia, S. Cahyawijaya, S. Min, M. Mozes, X. L. Li, I. Augenstein, A. Rogers, K. Cho, E. Grefenstette, L. Rimell, & C. Dyer (Eds.), Proceedings of the 7th Workshop on Representation Learning for NLP (pp. 1-8). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.repl4nlp-1.1

Valerio Miceli Barone, Antonio ; Birch, Alexandra ; Sennrich, Rico. / Distributionally Robust Recurrent Decoders with Random Network Distillation. Proceedings of the 7th Workshop on Representation Learning for NLP. editor / Spandana Gella ; He He ; Bodhisattwa Prasad Majumdar ; Burcu Can ; Eleonora Giunchiglia ; Samuel Cahyawijaya ; Sewon Min ; Maximillian Mozes ; Xiang Lorraine Li ; Isabelle Augenstein ; Anna Rogers ; Kyunghyun Cho ; Edward Grefenstette ; Laura Rimell ; Chris Dyer. Dublin, Ireland : Association for Computational Linguistics, 2022. pp. 1-8

@inproceedings{476fa13b0328421a8a114d0acc7ebdf2,

title = "Distributionally Robust Recurrent Decoders with Random Network Distillation",

abstract = "Neural machine learning models can successfully model language that is similar to their training distribution, but they are highly susceptible to degradation under distribution shift, which occurs in many practical applications when processing out-of-domain (OOD) text. This has been attributed to ``shortcut learning'''':'' relying on weak correlations over arbitrary large contexts. We propose a method based on OOD detection with Random Network Distillation to allow an autoregressive language model to automatically disregard OOD context during inference, smoothly transitioning towards a less expressive but more robust model as the data becomes more OOD, while retaining its full context capability when operating in-distribution. We apply our method to a GRU architecture, demonstrating improvements on multiple language modeling (LM) datasets.",

author = "{Valerio Miceli Barone}, Antonio and Alexandra Birch and Rico Sennrich",

year = "2022",

month = jun,

day = "3",

doi = "10.18653/v1/2022.repl4nlp-1.1",

language = "English",

pages = "1--8",

editor = "Spandana Gella and He He and Majumdar, {Bodhisattwa Prasad} and Burcu Can and Eleonora Giunchiglia and Samuel Cahyawijaya and Sewon Min and Maximillian Mozes and Li, {Xiang Lorraine} and Isabelle Augenstein and Anna Rogers and Kyunghyun Cho and Edward Grefenstette and Laura Rimell and Chris Dyer",

booktitle = "Proceedings of the 7th Workshop on Representation Learning for NLP",

publisher = "Association for Computational Linguistics",

note = "The 7th Workshop on Representation Learning for NLP, Repl4NLP 2022 ; Conference date: 26-05-2022 Through 26-05-2022",

url = "https://sites.google.com/view/repl4nlp2022/home",

}

Valerio Miceli Barone, A , Birch, A & Sennrich, R 2022, Distributionally Robust Recurrent Decoders with Random Network Distillation. in S Gella, H He, BP Majumdar, B Can, E Giunchiglia, S Cahyawijaya, S Min, M Mozes, XL Li, I Augenstein, A Rogers, K Cho, E Grefenstette, L Rimell & C Dyer (eds), Proceedings of the 7th Workshop on Representation Learning for NLP. Association for Computational Linguistics, Dublin, Ireland, pp. 1-8, The 7th Workshop on Representation Learning for NLP, Dublin, Ireland, 26/05/22. https://doi.org/10.18653/v1/2022.repl4nlp-1.1

Distributionally Robust Recurrent Decoders with Random Network Distillation. / Valerio Miceli Barone, Antonio ; Birch, Alexandra; Sennrich, Rico.
Proceedings of the 7th Workshop on Representation Learning for NLP. ed. / Spandana Gella; He He; Bodhisattwa Prasad Majumdar; Burcu Can; Eleonora Giunchiglia; Samuel Cahyawijaya; Sewon Min; Maximillian Mozes; Xiang Lorraine Li; Isabelle Augenstein; Anna Rogers; Kyunghyun Cho; Edward Grefenstette; Laura Rimell; Chris Dyer. Dublin, Ireland: Association for Computational Linguistics, 2022. p. 1-8.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Distributionally Robust Recurrent Decoders with Random Network Distillation

AU - Valerio Miceli Barone, Antonio

AU - Birch, Alexandra

AU - Sennrich, Rico

N1 - Conference code: 7

PY - 2022/6/3

Y1 - 2022/6/3

N2 - Neural machine learning models can successfully model language that is similar to their training distribution, but they are highly susceptible to degradation under distribution shift, which occurs in many practical applications when processing out-of-domain (OOD) text. This has been attributed to ``shortcut learning'''':'' relying on weak correlations over arbitrary large contexts. We propose a method based on OOD detection with Random Network Distillation to allow an autoregressive language model to automatically disregard OOD context during inference, smoothly transitioning towards a less expressive but more robust model as the data becomes more OOD, while retaining its full context capability when operating in-distribution. We apply our method to a GRU architecture, demonstrating improvements on multiple language modeling (LM) datasets.

AB - Neural machine learning models can successfully model language that is similar to their training distribution, but they are highly susceptible to degradation under distribution shift, which occurs in many practical applications when processing out-of-domain (OOD) text. This has been attributed to ``shortcut learning'''':'' relying on weak correlations over arbitrary large contexts. We propose a method based on OOD detection with Random Network Distillation to allow an autoregressive language model to automatically disregard OOD context during inference, smoothly transitioning towards a less expressive but more robust model as the data becomes more OOD, while retaining its full context capability when operating in-distribution. We apply our method to a GRU architecture, demonstrating improvements on multiple language modeling (LM) datasets.

U2 - 10.18653/v1/2022.repl4nlp-1.1

DO - 10.18653/v1/2022.repl4nlp-1.1

M3 - Conference contribution

SP - 1

EP - 8

BT - Proceedings of the 7th Workshop on Representation Learning for NLP

A2 - Gella, Spandana

A2 - He, He

A2 - Majumdar, Bodhisattwa Prasad

A2 - Can, Burcu

A2 - Giunchiglia, Eleonora

A2 - Cahyawijaya, Samuel

A2 - Min, Sewon

A2 - Mozes, Maximillian

A2 - Li, Xiang Lorraine

A2 - Augenstein, Isabelle

A2 - Rogers, Anna

A2 - Cho, Kyunghyun

A2 - Grefenstette, Edward

A2 - Rimell, Laura

A2 - Dyer, Chris

PB - Association for Computational Linguistics

CY - Dublin, Ireland

T2 - The 7th Workshop on Representation Learning for NLP

Y2 - 26 May 2022 through 26 May 2022

ER -

Valerio Miceli Barone A , Birch A, Sennrich R. Distributionally Robust Recurrent Decoders with Random Network Distillation. In Gella S, He H, Majumdar BP, Can B, Giunchiglia E, Cahyawijaya S, Min S, Mozes M, Li XL, Augenstein I, Rogers A, Cho K, Grefenstette E, Rimell L, Dyer C, editors, Proceedings of the 7th Workshop on Representation Learning for NLP. Dublin, Ireland: Association for Computational Linguistics. 2022. p. 1-8 doi: 10.18653/v1/2022.repl4nlp-1.1

Distributionally Robust Recurrent Decoders with Random Network Distillation

Abstract

Workshop

Access to Document

Fingerprint

Projects

Global Under-Resourced MEdia Translation

MTStretch: Low-resource Machine Translation

BroadSem-Induction of Broad-Coverage Semantic Parsers

Cite this