Group Distributionally Robust Knowledge Distillation

Konstantinos Vilouras*, Xiao Liu, Pedro Sanchez, Alison Q. O’Neil, Sotirios A. Tsaftaris

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Knowledge distillation enables fast and effective transfer of features learned from a bigger model to a smaller one. However, distillation objectives are susceptible to sub-population shifts, a common scenario in medical imaging analysis which refers to groups/domains of data that are underrepresented in the training set. For instance, training models on health data acquired from multiple scanners or hospitals can yield subpar performance for minority groups. In this paper, inspired by distributionally robust optimization (DRO) techniques, we address this shortcoming by proposing a group-aware distillation loss. During optimization, a set of weights is updated based on the per-group losses at a given iteration. This way, our method can dynamically focus on groups that have low performance during training. We empirically validate our method, GroupDistil on two benchmark datasets (natural images and cardiac MRIs) and show consistent improvement in terms of worst-group accuracy.

Original languageEnglish
Title of host publicationMachine Learning in Medical Imaging - 14th International Workshop, MLMI 2023, Held in Conjunction with MICCAI 2023, Proceedings
EditorsXiaohuan Cao, Xi Ouyang, Xuanang Xu, Islem Rekik, Zhiming Cui
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages9
ISBN (Print)9783031456756
Publication statusPublished - 15 Oct 2023
Event14th International Workshop on Machine Learning in Medical Imaging, MLMI 2023 - Vancouver, Canada
Duration: 8 Oct 20238 Oct 2023

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume14349 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference14th International Workshop on Machine Learning in Medical Imaging, MLMI 2023

Keywords / Materials (for Non-textual outputs)

  • Classification
  • Invariance
  • Knowledge Distillation
  • Sub-population Shift


Dive into the research topics of 'Group Distributionally Robust Knowledge Distillation'. Together they form a unique fingerprint.

Cite this