MEDFAIR: Benchmarking Fairness for Medical Imaging

Yongshuo Zong, Yongxin Yang, Timothy M Hospedales

Research output: Chapter in Book/Report/Conference proceedingConference contribution


A multitude of work has shown that machine learning-based medical diagnosis systems can be biased against certain subgroups of people. This has motivated a growing number of bias mitigation algorithms that aim to address fairness issues in machine learning. However, it is difficult to compare their effectiveness in medical imaging for two reasons. First, there is little consensus on the criteria to assess fairness. Second, existing bias mitigation algorithms are developed under different settings, e.g., datasets, model selection strategies, backbones, and fairness metrics, making a direct comparison and evaluation based on existing results impossible. In this work, we introduce MEDFAIR, a framework to benchmark the fairness of machine learning models for medical imaging. MEDFAIR covers eleven algorithms from various categories, ten datasets from different imaging modalities, and three model selection criteria. Through extensive experiments, we find that the under-studied issue of model selection criterion can have a significant impact on fairness outcomes; while in contrast, state-of-the-art bias mitigation algorithms do not significantly improve fairness outcomes over empirical risk minimization (ERM) in both in-distribution and out-of-distribution settings. We evaluate fairness from various perspectives and make recommendations for different medical application scenarios that require different ethical principles. Our framework provides a reproducible and easy-to-use entry point for the development and evaluation of future bias mitigation algorithms in deep learning. Code is available at
Original languageEnglish
Title of host publicationThe Eleventh International Conference on Learning Representations
Subtitle of host publicationICLR 2023
Number of pages38
Publication statusPublished - 1 May 2023
EventThe Eleventh International Conference on Learning Representations - Kigali, Rwanda
Duration: 1 May 20235 May 2023


ConferenceThe Eleventh International Conference on Learning Representations
Abbreviated titleICLR 2023
Internet address


Dive into the research topics of 'MEDFAIR: Benchmarking Fairness for Medical Imaging'. Together they form a unique fingerprint.

Cite this