Group Distributionally Robust Knowledge Distillation

Konstantinos Vilouras; Xiao Liu; Pedro Sanchez; Alison Q. O’Neil; Sotirios A. Tsaftaris

doi:10.1007/978-3-031-45676-3_24

Group Distributionally Robust Knowledge Distillation

Konstantinos Vilouras^*, Xiao Liu, Pedro Sanchez, Alison Q. O’Neil, Sotirios A. Tsaftaris

^*Corresponding author for this work

School of Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Knowledge distillation enables fast and effective transfer of features learned from a bigger model to a smaller one. However, distillation objectives are susceptible to sub-population shifts, a common scenario in medical imaging analysis which refers to groups/domains of data that are underrepresented in the training set. For instance, training models on health data acquired from multiple scanners or hospitals can yield subpar performance for minority groups. In this paper, inspired by distributionally robust optimization (DRO) techniques, we address this shortcoming by proposing a group-aware distillation loss. During optimization, a set of weights is updated based on the per-group losses at a given iteration. This way, our method can dynamically focus on groups that have low performance during training. We empirically validate our method, GroupDistil on two benchmark datasets (natural images and cardiac MRIs) and show consistent improvement in terms of worst-group accuracy.

Original language	English
Title of host publication	Machine Learning in Medical Imaging - 14th International Workshop, MLMI 2023, Held in Conjunction with MICCAI 2023, Proceedings
Editors	Xiaohuan Cao, Xi Ouyang, Xuanang Xu, Islem Rekik, Zhiming Cui
Publisher	Springer
Pages	234-242
Number of pages	9
ISBN (Print)	9783031456756
DOIs	https://doi.org/10.1007/978-3-031-45676-3_24
Publication status	Published - 15 Oct 2023
Event	14th International Workshop on Machine Learning in Medical Imaging, MLMI 2023 - Vancouver, Canada Duration: 8 Oct 2023 → 8 Oct 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14349 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	14th International Workshop on Machine Learning in Medical Imaging, MLMI 2023
Country/Territory	Canada
City	Vancouver
Period	8/10/23 → 8/10/23

Keywords / Materials (for Non-textual outputs)

Classification
Invariance
Knowledge Distillation
Sub-population Shift

Access to Document

10.1007/978-3-031-45676-3_24

https://arxiv.org/abs/2311.00476Licence: Creative Commons: Attribution (CC-BY)

Cite this

Vilouras, K., Liu, X., Sanchez, P., O’Neil, A. Q., & Tsaftaris, S. A. (2023). Group Distributionally Robust Knowledge Distillation. In X. Cao, X. Ouyang, X. Xu, I. Rekik, & Z. Cui (Eds.), Machine Learning in Medical Imaging - 14th International Workshop, MLMI 2023, Held in Conjunction with MICCAI 2023, Proceedings (pp. 234-242). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14349 LNCS). Springer. https://doi.org/10.1007/978-3-031-45676-3_24

Vilouras, Konstantinos ; Liu, Xiao ; Sanchez, Pedro et al. / Group Distributionally Robust Knowledge Distillation. Machine Learning in Medical Imaging - 14th International Workshop, MLMI 2023, Held in Conjunction with MICCAI 2023, Proceedings. editor / Xiaohuan Cao ; Xi Ouyang ; Xuanang Xu ; Islem Rekik ; Zhiming Cui. Springer, 2023. pp. 234-242 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{b56623740a1b4a86983ce6c92576283e,

title = "Group Distributionally Robust Knowledge Distillation",

abstract = "Knowledge distillation enables fast and effective transfer of features learned from a bigger model to a smaller one. However, distillation objectives are susceptible to sub-population shifts, a common scenario in medical imaging analysis which refers to groups/domains of data that are underrepresented in the training set. For instance, training models on health data acquired from multiple scanners or hospitals can yield subpar performance for minority groups. In this paper, inspired by distributionally robust optimization (DRO) techniques, we address this shortcoming by proposing a group-aware distillation loss. During optimization, a set of weights is updated based on the per-group losses at a given iteration. This way, our method can dynamically focus on groups that have low performance during training. We empirically validate our method, GroupDistil on two benchmark datasets (natural images and cardiac MRIs) and show consistent improvement in terms of worst-group accuracy.",

keywords = "Classification, Invariance, Knowledge Distillation, Sub-population Shift",

author = "Konstantinos Vilouras and Xiao Liu and Pedro Sanchez and O{\textquoteright}Neil, \{Alison Q.\} and Tsaftaris, \{Sotirios A.\}",

year = "2023",

month = oct,

day = "15",

doi = "10.1007/978-3-031-45676-3\_24",

language = "English",

isbn = "9783031456756",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "234--242",

editor = "Xiaohuan Cao and Xi Ouyang and Xuanang Xu and Islem Rekik and Zhiming Cui",

booktitle = "Machine Learning in Medical Imaging - 14th International Workshop, MLMI 2023, Held in Conjunction with MICCAI 2023, Proceedings",

address = "United Kingdom",

note = "14th International Workshop on Machine Learning in Medical Imaging, MLMI 2023 ; Conference date: 08-10-2023 Through 08-10-2023",

}

Vilouras, K, Liu, X, Sanchez, P, O’Neil, AQ & Tsaftaris, SA 2023, Group Distributionally Robust Knowledge Distillation. in X Cao, X Ouyang, X Xu, I Rekik & Z Cui (eds), Machine Learning in Medical Imaging - 14th International Workshop, MLMI 2023, Held in Conjunction with MICCAI 2023, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14349 LNCS, Springer, pp. 234-242, 14th International Workshop on Machine Learning in Medical Imaging, MLMI 2023, Vancouver, Canada, 8/10/23. https://doi.org/10.1007/978-3-031-45676-3_24

Group Distributionally Robust Knowledge Distillation. / Vilouras, Konstantinos; Liu, Xiao; Sanchez, Pedro et al.
Machine Learning in Medical Imaging - 14th International Workshop, MLMI 2023, Held in Conjunction with MICCAI 2023, Proceedings. ed. / Xiaohuan Cao; Xi Ouyang; Xuanang Xu; Islem Rekik; Zhiming Cui. Springer, 2023. p. 234-242 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14349 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Group Distributionally Robust Knowledge Distillation

AU - Vilouras, Konstantinos

AU - Liu, Xiao

AU - Sanchez, Pedro

AU - O’Neil, Alison Q.

AU - Tsaftaris, Sotirios A.

PY - 2023/10/15

Y1 - 2023/10/15

N2 - Knowledge distillation enables fast and effective transfer of features learned from a bigger model to a smaller one. However, distillation objectives are susceptible to sub-population shifts, a common scenario in medical imaging analysis which refers to groups/domains of data that are underrepresented in the training set. For instance, training models on health data acquired from multiple scanners or hospitals can yield subpar performance for minority groups. In this paper, inspired by distributionally robust optimization (DRO) techniques, we address this shortcoming by proposing a group-aware distillation loss. During optimization, a set of weights is updated based on the per-group losses at a given iteration. This way, our method can dynamically focus on groups that have low performance during training. We empirically validate our method, GroupDistil on two benchmark datasets (natural images and cardiac MRIs) and show consistent improvement in terms of worst-group accuracy.

AB - Knowledge distillation enables fast and effective transfer of features learned from a bigger model to a smaller one. However, distillation objectives are susceptible to sub-population shifts, a common scenario in medical imaging analysis which refers to groups/domains of data that are underrepresented in the training set. For instance, training models on health data acquired from multiple scanners or hospitals can yield subpar performance for minority groups. In this paper, inspired by distributionally robust optimization (DRO) techniques, we address this shortcoming by proposing a group-aware distillation loss. During optimization, a set of weights is updated based on the per-group losses at a given iteration. This way, our method can dynamically focus on groups that have low performance during training. We empirically validate our method, GroupDistil on two benchmark datasets (natural images and cardiac MRIs) and show consistent improvement in terms of worst-group accuracy.

KW - Classification

KW - Invariance

KW - Knowledge Distillation

KW - Sub-population Shift

UR - https://www.scopus.com/pages/publications/85176003671

U2 - 10.1007/978-3-031-45676-3_24

DO - 10.1007/978-3-031-45676-3_24

M3 - Conference contribution

AN - SCOPUS:85176003671

SN - 9783031456756

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 234

EP - 242

BT - Machine Learning in Medical Imaging - 14th International Workshop, MLMI 2023, Held in Conjunction with MICCAI 2023, Proceedings

A2 - Cao, Xiaohuan

A2 - Ouyang, Xi

A2 - Xu, Xuanang

A2 - Rekik, Islem

A2 - Cui, Zhiming

PB - Springer

T2 - 14th International Workshop on Machine Learning in Medical Imaging, MLMI 2023

Y2 - 8 October 2023 through 8 October 2023

ER -

Vilouras K, Liu X, Sanchez P, O’Neil AQ, Tsaftaris SA. Group Distributionally Robust Knowledge Distillation. In Cao X, Ouyang X, Xu X, Rekik I, Cui Z, editors, Machine Learning in Medical Imaging - 14th International Workshop, MLMI 2023, Held in Conjunction with MICCAI 2023, Proceedings. Springer. 2023. p. 234-242. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-45676-3_24

Group Distributionally Robust Knowledge Distillation

Abstract

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Other files and links

Fingerprint

Canon Medical / RAEng Senior Research Fellow in Healthcare AI

From trivial representations to learning concepts in AI by exploiting unique data

Cite this

Group Distributionally Robust Knowledge Distillation

Abstract

Publication series

Conference

Keywords / Materials (for Non-textual outputs)

Access to Document

Other files and links

Fingerprint

Projects

Canon Medical / RAEng Senior Research Fellow in Healthcare AI

From trivial representations to learning concepts in AI by exploiting unique data

Cite this