Loss Function Learning for Domain Generalization by Implicit Gradient

Boyan Gao, Henry Gouk, Yongxin Yang, Timothy M Hospedales

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Generalising robustly to distribution shift is a major challenge that is pervasive across most realworld applications of machine learning. A recent study highlighted that many advanced algorithms proposed to tackle such domain generalisation (DG) fail to outperform a properly tuned empirical risk minimisation (ERM) baseline. We take a different approach, and explore the impact of the ERM loss function on out-of-domain generalisation. In particular, we introduce a novel metalearning approach to loss function search based on implicit gradient. This enables us to discover a general purpose parametric loss function that provides a drop-in replacement for cross-entropy. Our loss can be used in standard training pipelines to efficiently train robust models using any neural architecture on new datasets. The results show that it clearly surpasses cross-entropy, enables simple ERM to outperform some more complicated prior DG methods, and provides excellent performance across a variety of DG benchmarks. Furthermore, unlike most existing DG approaches, our setup applies to the most practical setting of single-source domain generalisation, on which we show significant improvement.
Original languageEnglish
Title of host publicationProceedings of the 39th International Conference on Machine Learning
Number of pages15
Publication statusPublished - 23 Jul 2022
Event39th International Conference on Machine Learning - Baltimore, United States
Duration: 17 Jul 202223 Jul 2022
Conference number: 39

Publication series

NameProceedings of Machine Learning Research
PublisherMIT Press
ISSN (Electronic)2640-3498


Conference39th International Conference on Machine Learning
Abbreviated titleICML 2022
Country/TerritoryUnited States
Internet address


Dive into the research topics of 'Loss Function Learning for Domain Generalization by Implicit Gradient'. Together they form a unique fingerprint.

Cite this