Better Practices for Domain Adaptation

Linus Ericsson, Da Li, Timothy M Hospedales

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Distribution shifts are all too common in real-world applications of machine learning. Domain adaptation (DA) aims to address this by providing various frameworks for adapting models to the deployment data without using labels. However, the domain shift scenario raises a second more subtle challenge: the difficulty of performing hyperparameter optimisation (HPO) for these adaptation algorithms without access to a labelled validation set. The unclear validation protocol for DA has led to bad practices in the literature, such as performing HPO using the target test labels when, in real-world scenarios, they are not available. This has resulted in over-optimism about DA research progress compared to reality. In this paper, we analyse the state of DA when using good evaluation practice, by benchmarking a suite of candidate validation criteria and using them to assess popular adaptation algorithms. We show that there are challenges across all three branches of domain adaptation methodology including Unsupervised Domain Adaptation (UDA), Source-Free Domain Adaptation (SFDA), and Test Time Adaptation (TTA). While the results show that realistically achievable performance is often worse than expected, they also show that using proper validation splits is beneficial,as well as showing that some previously unexplored validation metrics provide the best options to date. Altogether, our improved practices covering data, training, validation and hyperparameter optimisation form a new rigorous pipeline to improve benchmarking, and hence research progress, within this important field going forward.
Original languageEnglish
Title of host publicationProceedings of the International Conference on Automated Machine Learning (AutoML 2023)
PublisherMIT Press
Pages1-25
Number of pages25
Volume228
Publication statusPublished - 2 Dec 2023
EventAutoML Conference 2023 - Berlin, Germany
Duration: 12 Sept 202315 Sept 2023
Conference number: 2
https://2023.automl.cc/calls/

Publication series

NameProceedings of Machine Learning Research
PublisherMIT Press
ISSN (Electronic)2640-3498

Conference

ConferenceAutoML Conference 2023
Country/TerritoryGermany
CityBerlin
Period12/09/2315/09/23
Internet address

Fingerprint

Dive into the research topics of 'Better Practices for Domain Adaptation'. Together they form a unique fingerprint.

Cite this