Learning to Segment from Scribbles using Multi-scale Adversarial Attention Gates

Gabriele Valvano, Andrea Leo, Sotirios A. Tsaftaris

Research output: Contribution to journalArticlepeer-review


Large, fine-grained image segmentation datasets, annotated at pixel-level, are difficult to obtain, particularly in medical imaging, where annotations also require expert knowledge. Weakly-supervised learning can train models by relying on weaker forms of annotation, such as scribbles. Here, we learn to segment using scribble annotations in an adversarial game. With unpaired segmentation masks, we train a multi-scale GAN to generate realistic segmentation masks at multiple resolutions, while we use scribbles to learn their correct position in the image. Central to the model’s success is a novel attention gating mechanism, which we condition with adversarial signals to act as a shape prior, resulting in better object localization at multiple scales. Subject to adversarial conditioning, the segmentor learns attention maps that are semantic, suppress the noisy activations outside the objects, and reduce the vanishing gradient problem in the deeper layers of the segmentor. We evaluated our model on several medical (ACDC, LVSC, CHAOS) and non-medical (PPSS) datasets, and we report performance levels matching those achieved by models trained with fully annotated segmentation masks. We also demonstrate extensions in a variety of settings: semi-supervised learning; combining multiple scribble sources (a crowdsourcing scenario) and multi-task learning (combining scribble and mask supervision). We release expert-made scribble annotations for the ACDC dataset, and the code used for the experiments, at https://vios-s.github.io/multiscale-adversarial-attention-gates.
Original languageEnglish
Pages (from-to)1-14
Number of pages14
JournalIEEE Transactions on Medical Imaging
Early online date30 Mar 2021
Publication statusE-pub ahead of print - 30 Mar 2021


  • Image Segmentation
  • Annotations
  • shape
  • Training
  • Logic gates
  • Semantics
  • Gallium nitride
  • Weak Supervision
  • Scribbles
  • Segmentation
  • GAN
  • Attention
  • Shape Priors


Dive into the research topics of 'Learning to Segment from Scribbles using Multi-scale Adversarial Attention Gates'. Together they form a unique fingerprint.

Cite this